Skip to content

Too Cool for Your Usability Test

CoolNo matter how well you recruit representative participants for a usability test and no matter how well you plan the testing, there are times when you’ll ask participants to perform a task that they might not normally perform themselves. It’s rare that every task you ask people to perform matches exactly what they would do. When this happens, most participants are agreeable enough to just “play along” for the purposes of the test.

Sure, it’s good to know what a participant would normally do instead of your planned task, but that’s more useful to learn during field studies. During a usability test, you usually just want to observe how well people can perform tasks.

During a recent usability test of an intranet design, I asked participants to browse the Blogs section to test out the usability of the filtering and searching functions common across the various sections of the intranet. I used the Blogs section as an example because that was the section we had built out in our prototype. Unfortunately, I came across two participants who were “too cool” to read blogs. In fact, they were too cool to even play along with my ridiculous and demeaning scenario.

It went something like this:

Me: Show me where you’d go if you wanted to see all the blogs in the company.

Joe Cool: Oh, I wouldn’t do that.

Me: Why?

Joe Cool: I don’t read the blogs.

Me: Why is that?

Joe Cool: Who cares about blogs? I don’t have time to read blogs.

Me: Okay, but if you did want to see all the blogs, where would you find them?

Joe Cool: I really wouldn’t do that. People here don’t really pay attention to the blogs. Who has time for that? We have enough to do with…

[Two minutes later]

Me: Okay, well that’s good to know, but just for the sake of this session, let’s say that you did want to read the blogs, where would you go to do that?

Joe Cool: [Sigh] Well, I guess I’d go here, and – here it is. But you see the problem with blogs is that…

[One minute rant later]

Me: Okay, what would you do if you wanted this to show you the most popular blog posts in the company?

Joe Cool: I don’t really care about what other people think is popular, especially from people who read blogs.

Me: Okay, but if someone else wanted to see the most popular blog posts in the company, what should they do here?

Joe Cool: Maybe they should ask someone else who reads blogs a lot? Or they should get a life and do something more productive.

Me: Okay, let’s move on to the next part…

Luckily, the next task was cool enough for him. Sometimes that’s all you can hope for.

How Many Brains is Your User Research In?

Brain

(Photo credit: Wikipedia)

Sure, it saves time to skip official research deliverables and just apply the research knowledge you’ve learned directly to the design, but it’s shortsighted. Unlike design, which leaves behind a tangible artifact that you can see, feel, and use; user research consists of abstract knowledge locked in the brain of the researcher until it’s presented effectively in a deliverable. If that knowledge isn’t officially captured and presented understandably to others, it tends to get lost over time.

That became clear to me recently when I was asked to take over a project for a Researcher who was leaving our company. Because of a very short timeframe, there was no official deliverable from her contextual inquiries, only informal notes. She was asked to give me a “knowledge transfer,” and provide me with her high-level notes. Needless to say, the knowledge transfer didn’t go very well, and she left the company taking her brain with her, including all the research it contained.

A similar mistake occurs when multiple researchers work on a project, dividing up the sessions between themselves. To project managers and clients, that can sound like a great idea. Two researchers can either get the research done in half the time, or they can double the number of participants in the same time frame. The problem is that instead of building up knowledge through repetition, each researcher only has the knowledge of half of the sessions. To analyze the results, they have to then somehow combine what they observed.

So until they invent a direct brain-to-brain transfer device: attend every session, don’t split up sessions between multiple researchers, and produce effective and comprehensive deliverables to document your findings for future generations.