The Importance of Pilot Tests
Weeknotes #29: Dec 7–11, 2020
This week I was preparing to kick off some unmoderated usability testing with TryMyUI. But before I pushed the button to kick off the tests, I ran a couple small pilot tests with one or two users.
These pilot tests were about ‘testing the test’, rather than focusing on the results. I’ll be honest, I’m not really used to doing pilot tests before my user research. When I’m running a 1:1 discovery interview with someone, I am a little loose with my scripts. If things go in an unexpected direction, I tend to go with it. As well, I’m actively engaged in the interview, so I can reframe or rephrase things if necessary.
But kicking off unmoderated usability testing is different. Since we are trying to measure things like “task success” and “time on task”, we want to make sure the instructions aren’t getting in the way, and we are being consistent in what we ask of people. We also can’t intervene while the test is going on. Basically since we won’t be there during the test, we need to put a little more time in up-front.
The pilot tests I ran were really valuable. The first participant spent nearly 15 minutes on the first task, and wasn’t actually able to complete the test in the time allotted. Of course you want to be careful about not drawing too many conclusions from an ‘n of 1’, but it drew my attention to things I hadn’t considered.
We also added in some guidance to participants that if they spent more than 4 or 5 minutes on one task, that they should state that they couldn’t find the answer, move onto the next task.
Because let’s be honest, when is the last time you spent more than five minutes on a website, searching for the answer to a question? You probably abandoned your search. We don’t want people to spend more time looking to complete a task during the test than we would realistically expect someone to in ‘real life’.
In the end, I removed a couple tasks and re-wrote another, and started recruiting for participants. I had the chance to start to listen to a couple of the interviews late on Friday, but I held off to really dig into research for this upcoming week, so I can really focus on it and things will be front-of-mind.
Can’t wait!
