Watch out for Survey McSurvface

If you want to improve your product’s documentation — or the whole user experience — there’s a tried and true technique: do a survey. At least that’s what we’ve always been told.

Let me tell you a couple of stories.

The boat


Come to think of it, “HMS Coke Can” might be a more suitable name. (Source: Natural Environment Research Council)

Earlier this week the British government, in the person of Science Minister Jo Johnson, announced that its new research vessel will not be christened Boaty McBoatface, even though that name won an Internet poll with 4 times as many votes as the runner-up.

Evoking memories of Graham Chapman’s Colonel, Johnson declared that the winning name was simply too silly and that a more “suitable” name will be chosen instead.

The bridge

Much the same thing happened in 2006 when Stephen Colbert, in his Comedy Central days, encouraged his viewers to vote in an online contest to name a bridge in Hungary.


The Almost-Colbert Bridge (Source: Wikimedia Commons / Civertan)

Stephen Colbert Bridge won, garnering more votes than there are people in Hungary. Things hit a snag when Hungary’s ambassador to the U.S. good-naturedly informed Colbert that in order to be honored, he would need to be (a) fluent in Hungarian and (b) dead.

Today the bridge is known as Megyeri Bridge because it connects two towns whose names end in -megyer. I’m not sure that’s better than Colbert Bridge. But I’m not Hungarian so I guess it’s none of my business.

The moral of both stories? Surveys and polls can be entertaining. But their results aren’t always useful.

Your customers

Now I know that nobody is going to turn your customer survey into a prank. Still, when you ask your customers what they want, they don’t always know. Their responses likely will be knee-jerk, not reflective of careful thought.

Want a better index? Sure, that sounds good. Bigger icons? Why not? Soon you’ve got a lot of “results” that you can turn into action plans. Yet you’ve missed the issues that truly affect the UX.

The solution? Don’t ask your customers what they want. Instead, ask them how they actually use the product, and ask them what things give them trouble. Do they have difficulty finding the instructions they need? Are the instructions relevant to their work situations? Are there product features that go unused because they’re hard to set up and maintain?

When you ask your customers how they really use your product, then you can use your own know-how to decide how best to make their lives easier.

There’s an even better way, although it’s harder than administering a survey. If you can observe your customers at work, if you can see for yourself where they succeed and where they struggle, then you’ll know exactly where to focus your efforts at improving both the documentation and the rest of the product.

So there you have it.

Surveys that ask customers what they want: too silly.

Surveys that measure the way customers actually use the product: much better.

In-person observation (including usability tests): harder, but best of all.

Tell me about experiences you’ve had improving your products by gathering information from your customers.

3 thoughts on “Watch out for Survey McSurvface

  1. Marcia Riefer Johnston

    Larry, “Boaty McBoatface” cracks me up. Points well taken.

    You ask for our stories. Here’s one that I told Scott Abel (as published in STC “Intercom” Nov/Dec 2012):

    “The importance of testing end-user instructions was impressed on me during my internship at Magnavox CATV, a manufacturer of cable TV network equipment. My fellow intern, Tim Voorheis, and I had just finished documenting how to use a cable TV box, at that time a new contraption. Tim and I had interviewed engineers, played with the device, written and illustrated all the steps, even color-coded the user tasks. We had created a mock-up of the printed piece we envisioned,including step-tabs that invited people to jump straight to the task they needed—the print equivalent of hotlinks. We had done everything we could think of to make this document foolproof. We had created, I was sure, a thing of beauty, something better than either of us could have created alone. A winner.

    “One lunch hour, our mentor, Karen, snagged a colleague for an impromptu test. I couldn’t wait to see the look of delight on our tester’s face. Karen handed the woman our color-coded mock-up. Tim and I watched (quietly, as instructed) from behind. With the instructions in one hand and the remote in the other, our tester dutifully pressed the correct buttons … and wondered why the cable TV box did not respond. How could we have anticipated that anyone would point the remote at the ceiling? Yet there she was, holding the remote straight up and down, moving her gaze back and forth from our pretty pages to the uncooperative remote.

    “In fact, we had illustrated the remote exactly that way—straight up and down. Our illustrations were two-dimensional, floating in white space with no context. It never occurred to us to tell people to aim the remote at the box. We had been blind to our assumption. Karen instantly knew what to do: revise our cover to show the remote pointed at the cable TV box.

    “Success! Without that simple 15-minute test, though, we would never have known that we had an information problem to solve.”


Tell me what you think

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s