Long distant usability test - a better experience than expected

This are a few notes about running usability tests at long distant. I never believed in it - but I think I have changed my mind.

As a final test of the new Customer website I created a usability test with help of XLS. I actually did not believe in this method but since we are running out of time. We needed to check up a few interface "problems" and we wanted to see if use right keywords is specified. So I set this test up reluctantly.

When you run a usability test sitting right beside the candidate you can tell them to give up. You can also see if they are so lost that they cannot have a chance to fix next question. You can adjust the language to the tester while running at volume test like this one you cannot.


You also have to make sure all the candidate is at right page while asking a question. Else it will be hard to compare the different results. Of course one some of the questions was totally free to perform in whatever the tester wanted to. I used a mixture.

I had three sections in my test:

1. New search interface where question were asked in a general way making the tester finding out his own search words. No one of the keywords used in question was used as keywords. For example - "you should order a meal for a patient" instead of "find the menue for patients"
2. The hierarchical structure actually containing all information. Previous interviews had shown me that about 50% of the users prefer hierarchical clicking to search. We had to keep that opportunity as well as A-Z.
3. A free part to see if the tester could find what he/she usually looks for at current customer web and/or the things your customer look for.

Of course I knew a few issues that could create problems. These issues I asked about several times but "disguised" not to bore the testers. I wanted to make sure they could manage those tasks nomatter what page they started from.

I also told the candidate not to work with any question longer than three minutes. And maximum time for all test was one hour.







The result
Today I got all results and it works much better than expected. The only thing I can complain about is that I didn´t lock the design of XLS file. Some tester had removed lines making it hard to compare the content.

I followed up all testers with a five minutes interview. What was hard? What was easy? How did you handle that problem with changing target group? Did you manage the auto complete function etc.

I think I will develop this test method further in next project. It was quick, it was pretty easy to achieve since I have lots of experiences running usability tests. The candidates had also similar background so the language was no problem. But I think I would have had serious problems if I sent the same test to doctors, nurses, cooks, laundry staff and administration staff. Different education, different background, different languages (yes we have got 35 within my organization) and IT knowledge would be a much to big problem.

I like the summary of this experience. I like to break down huge problems to small ones. Performing volume usability test take a lot of time and in my case I would need a lot of travelling. With XLS, email and 1.5 day of work I got a result that would have taken me a week with ordinary usability tests.
Why not try it!

---

It has to be fun, or else we cannot make it