The Testers Role
Before we start collecting data, testers are
given a warm-up script to familiarize themselves with the products and tasks
involved in the test. This step addresses the issue of learnability-how quickly
a new user can become proficient with a product.
In some tests, learnability is a key factor. It
may be crucial for evaluating products that have a worthwhile business purpose,
but are used infrequently. Learnability is also useful to measure how easily a
staff can be trained to be productive with a product. To reduce learnability as
a factor when measuring usability, we sometimes employ a mix of experience
levels-by including testers of varying experience with the products being
tested. This approach is a result of our premise that
"one-size-does-not-fit-all." In evaluating a group of products, we
include recommendations on which product is best for a specific task or a
particular type of user.
We
use a minimum of six testers for each project. To test products that appeal to a
wide variety of business users-i.e., word processors or spreadsheets-we double
that number to 12 for a more representative sampling. Every tester spends
between two and five hours with each product.
We
vary the order that testers work with the products in comparative reviews, where
more than one product is being tested, to keep the test fair. For example, if
"Tester A" works with "Product A" first, then "Tester
B" might work with "Product A" last. This helps to eliminate the
influence of fatigue as a variable, and allows all the products to evenly
benefit from the inevitable improvements in performance that occur as testers
learn the script.
Briefing Testers on Procedure
After we have chosen testers, more preparation
is needed before "live testing." First, testers are briefed and given
a chance to ask questions. The purpose of briefing them is to eliminate any fear
of failure. Most people are accustomed to tests where pass and fail scores
reflect on the person taking the test. Here, however, the test supervisor
clearly explains that it is the products, and not the people, that are on trial.
Then testers are given time to look over the
products being tested, their documentation and the tasks that will need to be
completed. They are allowed 15-20 minutes to familiarize themselves with the
product.
A tour of the lab is also part of the briefing.
This allows testers to become familiar with the facility, so it feels natural
when the test begins.
When questions have been answered, testers may
spend up to an hour with the product using a warm-up script. This exercise is
also important in making them feel more comfortable being in the usability lab.
It is critical to ensure that each tester's
experience in the usability lab is as close as possible to his or her everyday
working environment. The payoff is that it simulates the actual experience a
buyer will have with the product if he or she buys and uses it. It also allows
LULABS’ staff to evaluate the documentation and help-materials. More than
anything, though, it ensures that we are testing how easy the product is to use,
and not how quickly consumers can learn to use a new product-usability, not
learnability.
Talking During Test Encouraged
Another purpose of the briefing is to convince
testers to go against years of social conditioning and actually "think
aloud" during the test-the wireless microphones in the labs record valuable
information. But if testers do not speak up, pertinent data is lost. Testers are
encouraged over and over to talk their way through everything they do.
They are also requested to speak-up when
something strikes them as smart or helpful. Any time a manual or online help is
used, we want to know if it actually helped.
