Unexpected Complexity in a Traditional Usability Study
Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205
Article Contents
Conclusion
This study provides empirical evidence that validates Redish's observation that "Ease of use-what we typically focus on in usability testing-is critical but not sufficient for any product" (2007, p. 104). The ease of use indicators in this study, both quantitative and qualitative, suggested that our users found the prototype handbook and its visuals to be an "easy-to-use solution" to their perceived needs. And our client would very likely have been entirely satisfied with the ease of use data we provided. However, had the study's design solely focused on what the client told us they wanted, we would have failed both our client and the users. The true complexity of the tasks would not have been revealed, the lack of "utility" in the handbooks would not have manifested itself (Redish 2007), and our clients would never have seen the need to "think outside the box" of traditional handbook design. They would have lacked the cognitive dissonance that motivated at least some of the creative thinking that ultimately led to the successful handbook they ultimately published.
Fortunately, because we were working with clients who trusted us not to run up the cost of our study needlessly, we introduced some additional elements to the research methodology used. As a result of these additional elements, we observed the following points:
- It is important not to be swayed by clients and to ignore methodological triangulation. The decision to track the quality and correctness of the users’ performance was fortunate. Had it not been for the observation that users failed to have acceptable works cited entries, we would not have suspected that the problems were more complex than we had originally considered.
- Testing the market-leader and the prototype with the same scenarios helped the researchers to distinguish problems with information delivery techniques from confusion over the complexity of the tasks.
- Maintaining a singular purpose in visuals is important for success, even though complex information needs to be delivered. Use several visuals to convey different purposes.
- Users need to be made aware of the complexity of a problem before any potential solutions are introduced so they can look for additional contextual factors and use a recursive decision-making process.
- Modeling complex problem solving behaviors through the use of scenarios may lead to more effective performances from users when it’s simply not possible to capture or replicate all of the potential variables in a situational context (Flowers, Hayes, and Swarts, 1983).
As the field of usability studies grows and usability professionals are required to work with and adapt to the needs of new clients in new fields, the likelihood of encountering complex problems masquerading as simple ones will increase. Our research methods need to accommodate this and ensure that the studies we design for new clients include content experts' assessment of the quality of the users' performance and, when possible, head-to-head comparisons with competing products.
Acknowledgements
The author would like to thank Wendy Howard who collaborated on the data collection and analysis portions of this study. Thanks also belong to Alicia Hatter, Ginny Redish, and Barbara Mirel for the insightful feedback that was critical in producing the final version of this article.