Steve Sanderson

At Food on the Table, feedback from real users is the life-blood of how they do lean startup. When we needed qualatiative feedback from new users, one way was to hire people off of craigslist – a form of ad-hoc usability testing.

This question has come up so often, I promised to share the answer here:

Hi Steve:

I’ve got a question for you on doing usability testing via random folks on Craigslist.  Do you have an example of the ad that you guys use?  We’re going to shamelessly steal this concept, but we want to make sure we don’t write something that only attracts uber geeks.

The answer, abstracted a bit:

1. First, we need to pull together some information for the ad. Much of this should be derived from the experiment you’re working on (i.e. as part of the problem + hypothesis + experiment you’ve got – right?)
   1. what is the context of your experiment, e.g. “website” or “iphone app”
   2. what does your hypothesis + experiment say about how to filter or categorize the participants, e.g. for Food on the Table, we may have wanted to filter / categorize whether they cooked, whether they shopped, whether they used coupons. Turn these into individual questions about the participant.
2. More broadly, given your product, what is a few words associated with your product, e.g. for Food on the Table it may have been “recipes, meal planning and grocery shopping”
3. Now compose the ad:

title: <context> Usability Testers For Hire

body:

We are conducting a usability study, which means we would like your opinion about a new <context> we are developing that has to do with <few words associated with your product>.

The sessions last approximately an hour and pay $30.

Please answer the following questions with your response.

1. Name:

2. <questions about the participants>

…

12. Would you be available to do a research session at a downtown Austin office? Y/N

13. What day and time work best for you? (most sessions will take from 1-1.5 hours)

TIPS

• Often we had some additional questions in the email exchange – one of the big questions is “are you a developer” or similar; in our experience, developers make horrible usability testers – they want to critique the implementation, etc.
• We don’t include the name of our product or company because, as you may have guessed, we would like to guarantee that this is truly a new user, i.e. they don’t look ahead.
• Conducting these usability interviews is a distinct skill – without that skill the value of the learning is at risk. I hope to offer some specific tips on this later, but key points are:
  • don’t lead the witness – ask open-ended questions and wait… wait… wait…
  • watching what they do is as important as hearing what they say, i.e. body language communicates tremendous information – such as hesitation with a mouse, while verbally they’re assuring you it’s all fine.
  • try to get at least 2 people in the interview with the participant
  • get interviewers to sit around the participant so they can get different views
  • get all the interviewers to capture their thoughts during the process
  • don’t waste effort on trying to formalize results, instead debrief immediately afterwards with all the interviewers and the relevant other people in the company
  • everyone (yes, everyone) should participate as an interviewer regularly – developers, customers support, marketing, bus. dev, CEO – everyone.

Here’s a version of the ad we posted on craigslist:

Website Usability Testers For Hire

We are conducting a usability study, which means we would like your opinion about a new website we are developing that has to do with recipes, meal planning and grocery shopping

The sessions last approximately an hour and pay $30.

Please answer the following questions with your response.

 1. Name:

2. Age:

3. Do you do all/most of the cooking in the house? Y/N

4. Do you do all/most of the grocery shopping? Y/N

6. How many children under the age of 18 are in the house?

7. What grocery store do you go to buy your groceries?

8. How often do you use the Internet?

9. Would you be available to do a research session at a downtown Austin office? Y/N

10. What day and time work best for you? (most sessions will take from 1-1.5 hours)