Sometimes it’s ok not to ask ‘why’
Weeknotes #23: Oct 26–30, 2020
If you’ve read some of my other posts, you know I’m a fan of qualitative interviews and discovery, where the interviewee can attempt to uncover ‘why’ someone does something. This is in contrast to looking at behavioural analytics and focusing on the ‘what’, and ‘how often’.
In my new role at CRA, we perform a lot of unmoderated usability tests, with an aim to measure and report on task success and time-spent-on-task on our public-facing web content. The Government of Canada is pretty invested in the “Top Tasks” approach championed by Gerry McGovern, so this week I spent some time watching these masterclass videos from YouTube:
In one of the videos, Gerry mentions the importance of interviewees not guiding or influencing the research participant. That seems obvious, but it can be difficult, especially for newer researchers. I’ve observed plenty of research sessions where the researcher (intentionally or unintentionally) signals to the participant.
Gerry makes the case that you want the study to be as realistic as possible. And if the researcher is interjecting “why did you click there?” “what are you thinking?” “what do you think it means?” “did you notice…”— you are interrupting the person as they are just trying to find their way.
Unmoderated testing, where the person reads the instructions on their own and then works to complete the task, means that everyone gets the same instructions. Their results can be compared, because you don’t have to worry about slight inconsistencies. As well, the research participant may feel less conspicuous as they don’t have a real person watching them live. Gerry even suggests letting participants know that if they would abandon a task out of frustration were they not being observed, they should indicate that during the test. You don’t want people to keep struggling through to complete a task if they would not persevere under regular circumstances. By removing the live observer, maybe you end up with people being slightly more authentic. (Because we all know that people can be plenty candid when hiding behind a screen!)
Can people really explain ‘why’ anyway?
Another challenge to this utopian belief that we can just ask people why they performed a certain action, or what they were looking for, is that people don’t actually know all the reasons we do things and how we make decisions. For more on this idea, I recommend Danny Kahneman’s book Thinking: Fast and Slow. People are good at telling you what happened, but they’re generally pretty terrible at explaining why.
Not to mention, there can often be a disconnect where people tell you something was easy and made perfect sense.. but you clearly saw that they struggled or were unsuccessful. Last week I watched a study where participants scrolled up and down a page, navigating to different sections to try to find information.. that was in the first sentence of the content. They said it wasn’t difficult.. but just by watching, I knew that they struggled more than they should have to. It didn’t really matter if they accepted the amount of time the task took, we knew they weren’t performing the task as efficiently as we had expected. We didn’t have to ask them what they thought, we could just silently observe what they actually did (after the fact, watching a recording).
I wish the image above were flipped!
I love discovery, when you are uncovering possibilities. But there comes a time when decisions must be made, and measuring success in completing tasks is a great way to gain confidence. Are users able to complete the task they are attempting, in a reasonable timeframe? Is it a significant improvement over their current approach? If so, ship it! Rinse, repeat, continue to improve :-)
