And this is where my brain exploded

February 18, 2011

Half-way through a serious-minded survey on Internet use by a terribly reputable research provider, I reach this question.  Reproduced here in its entirety.

Which one of the following statements best describes how you use the internet to express opinions or share information with other people? Please select one only.

Quick, at the back, HOW MANY problems can you spot with this question?

1. Asking two questions in one

2. Giving twenty-five options

3. Many of the options don’t relate to the questions (how can ‘listening to a podcast’  be classified as ‘sharing’ or ‘expressing an opinion’??)

4. Let’s just say that this reveals a very peculiar mental model of internet use

5. Twenty-five options.

The memory load is just astounding – you have to hold 2 questions and 25 options in your mind.  I had forgotten the first couple by the time I got to the middle.

What’s more, it’s not that my memory is failing.  My utter lack of recall is supported by science.

While ‘the magical number seven (plus or minus two)’ isn’t an infallible guide to the number of options you can give people, it’s probably nearer the truth.  The fewer options the better. If you have to list more, it’s …well, it’s probably because your question sucks.

There is also Hick’s law which states that the more items you have to choose from (as with items on a website menu), the longer it takes to make a decision.

You can reduce the memory load to some extent  by organising the information into related chunks (for example, putting all the podcast options together). You can also order the list meaningfully to make it easier to scan (alphabetical order, subject order).

But mostly you need to draw a big line through the question and start again.

Email This Page

{ 9 comments… read them below or add one }

Tom Webster February 18, 2011 at 2:00 pm

This made my head hurt.


Anonymous February 18, 2011 at 2:10 pm

It is actually unanswerable, isn’t it? I think I bombed out at that point, after my brain ran in little circles for a while.


Conversition February 18, 2011 at 2:27 pm

pardon me while I throw up. Given that the checkbox let’s me choose more than one best option, i’m going to select “climbing trees” because it makes just as much sense.


Anonymous February 18, 2011 at 2:40 pm

Oh, I didn’t mention that. Here have a cookie!

It’s ‘choose one only.’ But with a checkbox, not a radio button (I tried).

This was a big company, too.


Jpuleston February 18, 2011 at 4:31 pm

Alison do you want some more open submissions to your hall of shame! I collect them too.


Anonymous February 18, 2011 at 7:03 pm

I think we all do. You are the only person I know who has awesome, adorable questions.


Claire Tompkins March 22, 2011 at 3:04 am

I forgot what the question was by option 8 or 9. I’ll be most of their respondents took one glance at that list and went, nuh-uh, thanks.


Nik Sargent March 23, 2011 at 11:25 am

most questionnaires suck by definition, because they enforce the world-view and pre-defined expectations of the questioner on the answerer. As such you tend to get what you ask for, not necessarily new insight. This is just a beautiful example of such horror.


Ali Mac March 23, 2011 at 1:48 pm

‘they enforce the world-view and pre-defined expectations of the questioner on the answerer’

I think that’s exactly it – the underlying question (or questions!) is hugely flawed. Even if you cut it down to ‘express opinions’, I’m damned if I could decide between blogging, Twitter or all the other stuff. What they get out will be nonsense.


Leave a Comment

{ 1 trackback }

Previous post:

Next post: