I published a new article in UXmatters this week, “What Could Possibly Go Wrong? The Biggest Mistakes in Usability Testing.”
This article came out of thinking about all of the mistakes I’ve made, and problems I’ve encountered, over the last 16 years conducting usability testing. I think it’s good to look back and think about the lessons you’ve learned. This article is jam-packed with advice learned the hard way.
Usability testing is the most highly structured user research method. Compared to field studies and interviews, the tasks and questions are usually highly planned, and you usually stick pretty close to the discussion guide. That also makes it the most repetitive method. You see the same types of people performing the same tasks and answering the same questions over and over again.
After you get some experience, you can begin to think of usability testing as routine and pretty easy. At a former company, it was the first task that we gave to new researchers, just out of college. It seemed like the easiest method to learn. That may be true, but there are still all kinds of mistakes that can occur. This article discusses the main problems and how to avoid them.
Photo by Blue Oxen Associates on Flickr
I just published an article on UXmatters, 10 User Research Myths and Misconceptions. It addresses common misunderstandings about user research that I’ve encountered over the years.
Here’s a bonus outtake from the article, Myth 11…
Myth 11: Field Research Is Better Than Usability Testing
On the other end of the spectrum from those who don’t understand the difference between user research and usability testing, are the user research elitists who think up-front, generative user research methods are far superior to usability testing. In this view, field studies take researchers out of the lab to observe people in their natural environments performing their usual activities, while usability testing takes place in the sterile, artificial environment of a usability lab and asks people to perform a limited set of artificial tasks. Instead of learning about people and what they really do, usability testing provides the limited value of learning whether people can perform your artificial tasks.
The Truth: Both Field Research and Usability Testing Have Their Places
Field studies and usability testing are two different methods used for different, but equally important, purposes. Field studies provide information to inform design, while usability testing evaluates a design. You have to make interpretations and conclusions from the user research and apply that to a design. Even after very thorough user research, you’re never completely sure that what you’ve designed will work well for the users. Usability testing is the evaluation that either confirms your decisions or points you to refinements. Both user research and usability testing are important and necessary. There’s no reason we can’t appreciate the value of both methods.
Analyzing the data is the most interesting part of user research. That’s where you see the trends, spot insights, and make conclusions. It’s where all the work comes together and you get the answers to your questions.
Why, then, did I publish an article in UXmatters – Analysis Isn’t Cool? All too often I’ve realized that clients, management, and project stakeholders underestimate the analysis phase and just want to get to the answers. People like to say that they did user research, but they don’t like to spend the time to analyze the data. They like the deliverables, whether they read them or not, but they don’t want to spend a lot of time on the analysis to produce those deliverables.
In this article, I discuss what analysis involves, methods for individual and group analysis, and ways to speed up the analysis process.
Photo by Josh Evnin on Flickr
I’m happy to report that user experience is alive and well in Bulgaria. I just got back a week ago from attending and presenting at UXify 2015 in Sofia, Bulgaria. A very enthusiastic audience of over 300 people attended a day of presentations, followed by a day of in-depth workshops. UXify was the end of a month of user experience events – UX Month Sofia.
On Friday, June 19th, I presented User Research in the Wild, a presentation about visiting people in their natural environments to observe their tasks. In addition to being a how-to guide, I covered common problems you face in conducting field studies and how to solve them, and how to overcome obstacles to field research.
On Saturday, June 20th, I gave a workshop on Paper Prototyping to about 20 attendees. After an introduction to creating paper prototypes and how to test them, the participants divided into groups to create their own paper prototypes. We only had a limited time to create them (about 40 minutes), so I was very surprised by how detailed and creative their prototypes turned out in such a short period of time. Each group of two people then joined with another group to take turns conducting usability testing on their paper prototypes. Each group did four rounds of tests, switching roles, so that each person was able to experience the role of the facilitator, “the computer,” and the participant twice. The attendees really seemed to have a good time, with a lot of laughing and joking around.
In between my own presentations, I attended several interesting presentations about dashboard design, gamification, socially responsible design, and ecommerce user experience.
It was a fun trip, and it was nice to meet a lot of similar people, in another culture, who are similarly committed and enthusiastic about user experience.
I recently wrote an article in UXmatters about how the information displays in my Prius changed my driving behavior. Seeing the miles per gallon (MPG) I was getting in real time, acted like a feedback loop, causing me to make a connection between my driving actions and how they affected my gas mileage.
Something I didn’t write about, however, was another type of feedback that you get as a Prius driver – the drivers behind you. When someone drives up behind you and tailgates, that’s feedback to go faster or get out of the way. However, that feedback conflicts with the feedback from your Prius to drive economically.
To drive economically and get good gas mileage requires a steady pace and slow acceleration. That doesn’t necessarily mean slow driving. I drive 65 to 75 miles-per-hour to work, while getting about 60 miles per gallon. But it does mean that you need to avoid quickly accelerating. So being a Prius driver sometimes puts your gas-saving goals in conflict with the drive-fast goals of other drivers. There’s a reason some drivers get annoyed by Prius drivers.
Although I always drive in the right hand or middle lanes when I’m driving economically, I’ve found that the best way to drive economically is to ride along behind someone else. If there’s no one in front of you, and another driver comes up from behind, you feel pressure to go faster. If you have another driver in front of you, you have an excuse to drive economically. It’s that guy’s fault, you’re just driving behind him at a reasonable distance. So you don’t feel the pressure to speed up.
I’ve gotten my highest MPG records from driving (at a safe distance) behind a truck that was going a steady, reasonable speed (maybe 70 MPH). That allows you to ignore the feedback from drivers behind you and pay attention to the MPG display.
Sounds like I’ve put too much thought into this gas mileage thing with my Prius. Maybe, but I save money and it’s fun.
Ugh! It’s December, and I feel I have to write something. I haven’t published anything since November, and I feel like I should have at least one post a month. But it’s hard to write a blog. Finding the time to write something worthwhile is difficult. When you also publish elsewhere, as I do, it’s especially difficult. Your blog ends up as the recipient of whatever energy and ideas are left after publishing elsewhere.
Several recent developments have also made me realize how difficult it is to publish a Web magazine. These magazines are run by volunteers and are rarely money making ventures. They depend on volunteer writers and editors, who each have their own work and personal responsibilities competing with their side project – working on the Web magazine.
Last week, for example, Johnny Holland, one of the leading Web magazines focused on interaction design, announced that it was ending its run (at least for now). In November, Boxes and Arrows, perhaps the oldest UX Web magazine, announced that it was Not Dead Yet and asked for the next generation of volunteers to take over running the site.
There are still several great UX Web magazines out there. I highlight these in my latest article on UXmatters – Publishing and Presenting, Part 2: Publishing. If you’ve ever had an interest in writing or getting involved in publishing, I recommend reading my advice about the writing process and where and how to get published.
If you need motivation to start publishing or presenting, check out part one – Publishing and Presenting, Part 1: Yes, You Can! And look for part 3 coming out in the next few weeks, about presenting at conferences.
I just got back from the IA Summit 2012 in New Orleans, where I presented User Research is Unnatural (But That’s Okay). Yes, that may seem like a strange topic for a user researcher to present, but I think it’s very important to remember how strange and unnatural user research can seem for participants. The point of my presentation is that by remaining aware of the awkward and uncomfortable aspects of user research, we can take steps to minimize or eliminate those problems to get better research results.
See my slides on Slideshare: User Research is Unnatural (But That’s Okay)
This was my first time attending the IA Summit, and it was a great conference with some excellent speakers and intriguing topics. As a user researcher who more often attends UPA and CHI conferences, it was interesting to attend a more design-oriented conference. I’m already looking forward to next year’s IA Summit, which will be in Baltimore.