Skip to content

Two Rarely Used Research Methods

Observing in a public areaMy two most recent articles in UXmatters have been about two research techniques that are very common to anthropology and the social sciences but are rarely used in design research. Covert naturalistic observation and participant observation certainly require more work and time than we usually have in UX projects, but it’s worth taking a look at these two methods to see what we can adapt to design research.

Covert Naturalistic Observation
This type of study is known in psychology, anthropology, and other social sciences as covert naturalistic observation. It is the opposite of the techniques we typically use, which are forms of overt naturalistic observation. Being covert means observing behaviors in their natural contexts without any intervention or influence by the researcher and without participants knowing that they’re being observed.

Participatory Observation
Anthropologists and sociologists often practice participant observation, in which they join a group as a participating member to get a first-hand perspective of the group and their activities. Instead of observing as an outsider, they play two roles at once—objective observer and subjective participant…

UX Testing?!!

Old Man Usability

Old Man Usability

Okay, now wait just a goddamn minute! UX testing? U-X testing?!! Now that’s just going too far!

You think you’re all better than me and don’t need “usability” anymore? “User experience” is a more inclusive and descriptive term about the aspects we’re interested in these days. Yeah yeah, fine. It’s more than just usability. Okay, I get it.

But keep your damn UX hands off my usability testing!!! That’s my signature method. I invented that! Talk about kicking a man when he’s down.

What am I talking about, you say? I’ve begun to notice this disturbing trend of you UX creeps stealing my method and calling it “UX” testing. Just look at this recent article from those fancy-pants, “digital marketers” over at eConsultancy: A Case for UX Testing and Agile. And then I noticed this article from last year: UX Testing and Cultural Preferences. Even User Zoom has gotten into the act with this article: 17 Questions Answered About UX Testing and Agile. And it doesn’t stop there. I just Googled “ux testing” and got 28,300 results!

Usability testing has been providing more than just usability for a long time. So in some ways I see your point that perhaps the word “usability” only describes part of what this method provides insight into.

But usability testing is the one method that’s still primarily about usability. Put participants in a lab (or test them remotely), give them tasks to perform, observe their behavior, and ask them to tell you what they’re thinking – that’s usability testing. In addition to assessing usability, it can provide information about satisfaction, emotions, and opinions, but it doesn’t give you a true depiction of the user experience. Other UX research methods give you a better picture of the user experience by observing people in their natural contexts of use. You can test usability, but you can’t really test the user experience.

And what are these people who are doing “UX testing” really doing? You guessed it! Usability testing. It’s nothing different. Just a name change.

So, I agree that user experience makes sense, but that doesn’t mean you should do a global find and replace, turning every instance of “usability” into “user experience.”

So keep your damn hands off my usability testing! It will always be “usability testing” no matter what you want to call it.

By the way, Googling “usability testing” brings up 2,110,000 results. So there!

– Old Man Usability

How is usability testing like beer?

Glass of beer

Flickr: HeadCRasher

My manager asked me recently, “Do you think usability testing has become a commodity?” He was referring to the fact that in the last year or two we’ve seen clients go with cheaper usability testing companies. He was questioning whether clients have decided that there’s no difference between usability testing companies except price. Quality isn’t a differentiator to them anymore (if it ever was).

It does seem that there are companies out there doing usability testing at increasingly lower prices. How they cut corners to get their costs low enough to still make a profit amazes me, and it makes me wonder what kind of quality the clients receive.

Then an analogy dawned on me – usability testing, as a consulting service, is like beer. There are many people out there who are perfectly happy drinking cheap beer. It’s cheap, it’s bland, but it does the trick in the end – it gives you a buzz. But that doesn’t mean everyone is satisfied with cheap beer. There are still those out there who appreciate and will pay more for craft beer with quality, taste, and a better buzz.

If you want cheap usability testing, you can get it. It won’t taste good or be the best quality, but if all you’re looking for is a cheap buzz, it will do the trick in the end. On the other hand, if you have a sophisticated enough palette, you’ll be able to tell the difference between cheap usability testing and craft usability testing, and you’ll be more satisfied in the end. The bottom line is: you get what you pay for. There’s usability testing out there for all tastes and budgets.

Too Cool for Your Usability Test

CoolNo matter how well you recruit representative participants for a usability test and no matter how well you plan the testing, there are times when you’ll ask participants to perform a task that they might not normally perform themselves. It’s rare that every task you ask people to perform matches exactly what they would do. When this happens, most participants are agreeable enough to just “play along” for the purposes of the test.

Sure, it’s good to know what a participant would normally do instead of your planned task, but that’s more useful to learn during field studies. During a usability test, you usually just want to observe how well people can perform tasks.

During a recent usability test of an intranet design, I asked participants to browse the Blogs section to test out the usability of the filtering and searching functions common across the various sections of the intranet. I used the Blogs section as an example because that was the section we had built out in our prototype. Unfortunately, I came across two participants who were “too cool” to read blogs. In fact, they were too cool to even play along with my ridiculous and demeaning scenario.

It went something like this:

Me: Show me where you’d go if you wanted to see all the blogs in the company.

Joe Cool: Oh, I wouldn’t do that.

Me: Why?

Joe Cool: I don’t read the blogs.

Me: Why is that?

Joe Cool: Who cares about blogs? I don’t have time to read blogs.

Me: Okay, but if you did want to see all the blogs, where would you find them?

Joe Cool: I really wouldn’t do that. People here don’t really pay attention to the blogs. Who has time for that? We have enough to do with…

[Two minutes later]

Me: Okay, well that’s good to know, but just for the sake of this session, let’s say that you did want to read the blogs, where would you go to do that?

Joe Cool: [Sigh] Well, I guess I’d go here, and – here it is. But you see the problem with blogs is that…

[One minute rant later]

Me: Okay, what would you do if you wanted this to show you the most popular blog posts in the company?

Joe Cool: I don’t really care about what other people think is popular, especially from people who read blogs.

Me: Okay, but if someone else wanted to see the most popular blog posts in the company, what should they do here?

Joe Cool: Maybe they should ask someone else who reads blogs a lot? Or they should get a life and do something more productive.

Me: Okay, let’s move on to the next part…

Luckily, the next task was cool enough for him. Sometimes that’s all you can hope for.

What’s Wrong With Usability Anyway?

old man

Old Man Usability

 

Okay, fine. I get it. You don’t think that I, usability, am cool anymore, and you don’t want to be seen with me now. I’m the dorky, embarassing parent, and you want to hang out with your cool, “user experience” friends. That’s okay. It’s only a natural part of growing up, I guess.

 

Although I brought you into this field, gave you your first job, supported you, and brought you respect and recognition, I guess you’re ready to go out on your own now, and you need to establish your own identity. That’s understandable, but I must admit I was a little hurt when even my most loyal child, the Usability Professionals Association changed its name to the User Experience Professionals Association. Okay, actually that one hurt a lot.

Don’t get me wrong, I do admit that “user experience” makes sense. There’s more to what people experience than just usability. I realize that. But don’t ever forget that usability is still very important. In fact, I’m probably the most important of the elements that make up user experience. If something isn’t usable, then it can’t really be useful, desirable, or valuable can it?

In fact, most of what you and your friends do under the name “user experience” today is what we did back in my day, under the name “usability.” So I don’t really see the big difference.

I think I deserve a little respect, though. I spent many years making a name for myself and getting people to think about the needs of the user. The current popularity of user experience wouldn’t be possible without the trail I blazed first. At least people know my name, usability, and what it means. Try finding a consistent definition of user experience, ha!

So after all I’ve done for you, this is the thanks I get? People declare that usability is old, tired, boring, uncool, not innovative, and even claim that I’m dead? Just wait until you have offspring of your own. See how you feel when they move on from user experience to the next buzz word.

Already I can see it beginning. Everyone’s jumping on the bandwagon and calling themselves a user experience professional these days. The term user experience is getting too broadly defined and overexposed. I can feel the pendulum starting to swing back. At this point, I’m so uncool that I’m actually becoming cool again. Soon I’ll be able to say, “I’m back!” Just you wait and see!

Tattle Tale Participants

Have you ever come across a tattle-tale participant? See if this sounds familiar.

You conduct a user research session (interview, contextual inquiry, focus group, etc.), asking simple questions to get the participants to speak about the subject at a basic level of understanding. Although you may know some of the answers to the questions you’re asking, you ask the questions anyway to hear the answers from the participants’ perspective. Misunderstanding your intent, the participant is alarmed that you don’t know what you’re doing and tattles on you to the client, “It was clear from his questions that he didn’t know anything about our process/system/technology.”

For example, a few years ago I was doing research on a company’s use of SAP. We had a group of interns show us the HR tasks that they do in SAP. Our questions led them to conclude that we didn’t know anything about SAP (which was mostly correct). They tattled to their manager who contacted our main client, alarmed that we weren’t SAP experts. Fortunately, he reassured her that we were user experience experts, and we weren’t supposed to be SAP experts.

What turns a participant into a tattle tale? Usually, it comes from a misunderstanding of our role. As user experience consultants working on many different projects, we constantly have to learn about new organizations, new systems, new processes, new technologies, and new types of people. We interview business stakeholders and conduct user research to learn about these things, but our goal isn’t to become experts. In fact, it’s often better to not be an expert. We have the advantage of seeing a group, system, process, or set of tasks from an outside perspective. Because we don’t already have the same insider knowledge as the business stakeholders and participants,  we can get them to explain things to us as outsiders. To do this, we often need to ask basic questions and sometimes even act “dumb” to get participants to fully explain things that they would otherwise forget to explain or gloss over at a high level.

The worst thing about tattling is that it can make us afraid to ask questions for fear that they might expose our “ignorance.” So make it clear who the experts are – the users and the business stakeholders, and where your expertise lies – user experience. Combining the expertise of business stakeholders, users, and user experience professionals is the key to a successful project.

New Article: Capturing User Research

My latest article for UXmatters is about Capturing User Research. It discusses the pros and cons of various methods of capturing user research from handwritten notes, typing up notes on a laptop or tablet, having someone else take notes, recording the audio, recording with video, taking photos, logging, and simply relying on your memory. 

Read the entire article at UXmatters: Capturing User Research.