Insights

To Use Remote or Moderated User Testing, that is the Question!

Blog / 05 Dec 2016

Tags: Usability Testing / User Experience / UX / UX Research

So you’re ready to run some User Testing? Great. What’s that you say? You’re not sure whether to get users into the lab or use remote testing software? No problem, let us help you out with that…

First of all, let’s be clear – watching any ‘real’ users interact with your site or app is better than not doing it at all, regardless of whether you use a lab or not. It’s all too easy to get buried in the day-to-day running of a site and forget what the process of browsing or buying is really like for your customers. Watching them be honest about the experience of interacting with your site never fails to be enlightening, though you may not like everything you hear…

First of all, let’s be clear – watching any ‘real’ users interact with your site or app is better than not doing it at all, regardless of whether you use a lab or not. It’s all too easy to get buried in the day-to-day running of a site and forget what the process of browsing or buying is really like for your customers. Watching them be honest about the experience of interacting with your site never fails to be enlightening, though you may not like everything you hear…

Sometimes the easy route isn’t the right route

Remote testing is popular and increasingly easy to do. It can look quicker and cheaper, but sometimes that’s not the case. What remote testing is great for is letting you see how users in a wide variety of locations use your site – you can get users from around the globe to complete the same task if needs be. Remote testing companies usually have quick and easy access to a pool of users, so it can be possible to have videos to analyse in less than a day.

However, there are drawbacks to remote testing and many of these relate to the lack of a moderator. Ok I guess I am a little biased, but a good moderator is worth their weight in gold (well, perhaps tea and biscuits) when it comes to getting insights from customers. For example, an unmoderated test relies on the user understanding the task and staying on track. A simple misunderstanding of the task, for example misreading ‘smart casual’ as ’smart’ during a clothes shopping task, can prejudice a user towards the products on a site and render their feedback unusable. Remote testing users also have to run the test themselves and can therefore be more focused on completing the test than the task you’re interested in. In lab based testing both of these issues can quickly be resolved by a well-timed intervention from a moderator, meaning that the session isn’t wasted.

Don’t underestimate the power of body language

Even moderated remote testing cannot gather the same level of data that you can achieve by having someone in the room as the user goes through your site. Body language is an important way we communicate that is not possible to analysis in remote testing. A good moderator is also able to ask questions of the user, delving deeper into issues that remote testing can only skim the surface of. By keeping the questions non-leading and timing them to avoid distracting the participant, a moderator can uncover the user’s motivations, their expectations and any gaps between these and what the site is delivering.

Silence is golden

Sometimes the benefit of a moderator is that they know when not to talk. Remote sessions rely on users following the ‘think aloud’ protocol to explain their behaviour as they complete a task. Have you ever tried to explain how to kick a ball as you do it? The simple act of explaining something often makes it harder to do. In the lab we frequently allow users to complete tasks in silence if they prefer, so they can really focus on what they’re trying to do. Once they’ve finished we revisit the journey with them, asking what they thought and why they made certain decisions. This allows them to analyse their own behaviour without it changing what they naturally do.

Be a part of the testing

For our clients, another massive draw for lab testing is that they can come and watch, live, as the sessions happen. This means they’re able to talk to the moderator during the day and tweak the sessions to get the most out of them. Refining the task questions to get better answers out of later participants is really helpful and cannot be done after receiving a batch of remote testing videos.

Finally, when your sessions are all complete, the biggest difference is analysis. You may have your remote user videos but who is going to watch them and produce actionable insights from them? With moderated testing the moderator is able to give insights right from the first session. We prefer to turn those insights into a report and a face-to-face debrief but, regardless of the deliverables, simply recording users won’t give you analysis, insights and next steps.

So hopefully that’s helped you understand the advantages and disadvantages of remote and lab testing. If you’re looking for a handy takeaway, the below table sums it all up. But if you’re still not sure which route to go down for your specific project get in contact here!