bad surveys

I’m a fond member of the blogging/social networking site, Livejournal.   Over the last few days, I’ve seen the most incredible shitstorm unfold, over the cack-handed efforts of two rogue academics to research what they were pleased to call ‘the cognitive neuroscience of fanfiction’.

Background

First, a bit of background: Livejournal (one of the original social networks) is a vast and varied set of subcultures, and interconnected blogs, dominated by film, TV, book and gaming fans.    It is more counterculture than culture, really: it tends to be left-wing, creative and anarchic.

One of the many subcultures in the mix is fanfiction writing:  stories that people write using characters from books, film, music and TV.  Fanfic writing is female-dominated, and some of it (but by no means all) is very explicit.   There is fanfic for everything, from Jane Austen through Doctor Who (rewriting the works of Russell T. Davies) to The Mighty Boosh.

Fanfic writers have an odd hobby, but they are a pleasant and literate bunch who are much studied by academics.   In fact, academics (like Henry Jenkins) completely adore this stuff  – it pulls feminism, transgression, social networking and copyright laws all into one place. What’s not to like.

The questionnaire is launched

Anyway, a few days ago a friend forwarded me a link to an online questionnaire that she found intriguing.  It was about fanfiction, it seemed a bit amateur, and what did I think of it?   The link was banner-style, and it looked a lot like the Cosmo-style pop quizzes that are memed all over the place on social networks.   There was a reassuring link to a FAQ page giving the names of the researchers, Ogi Ogas and Sai Gaddam, and their academic affiliations at Boston University (the BU links no longer exist).  This page also gave a long explanation of their interests in cognitive neuroscience, and what this had to do with fanfiction…

“We’re deeply interested in broad-based behavioral data that involves romantic or erotic cognition and evinces a clear distinction between men and women. Fan fiction matches this criteria perfectly.”

…Uhuh.

The researchers had apparently also consulted a couple of well-known bloggers in the areas, and got their guidance and feedback and endorsement.    Apart from the fact that the academics weren’t making the changes suggested, it all seemed fine.

The online questionnaire itself (captured here in two parts on an LJ Dr Who site – you may see an generic age warning for content on some LJ pages) was a rather different story. I took a look.  There were  70 questions in all (one per page), starting with some brusque questions about one’s gender, age and ethnicity.   It even asked for your SAT scores.  The questionnaire proceeded to a number of fantastically detailed and rather odd questions dealing with fanfiction reading habits; and then it got heavily intimate, asking (amongst other things), exactly what kinds of sexy stories the respondent read and (deep intake of breath) whether they ever had rape fantasies.

The questionnaire… does not go down well.

The questionnaire was barely up before LJers started complaining about the content.  LJ people love to complain at the best of times, and there was a lot of ground to cover here:

a)      Terrible questionnaire design

b)      Inaccurate, amateurish and homophobic wording

c)      Prurient lines of questioning

d)      No attempt to screen out under-18s

e)      Lack of the usual information on privacy, anonymity and confidentiality

f)       And (my favourite) frequent criticisms of the methodology.  How in the name of heaven the researchers were going to draw any valid conclusions whatsoever about subcortical processing, given their data collection methods?

What the researchers hadn’t bargained for was the thoughtfulness of the response.  Livejournal people are a fairly literate bunch.   Stuff like feminist analysis of television casting decisions is a walk in the park for many of them.   At least some of the people who came across the questionnaire were social researchers, lecturers, feminist academics, and indeed neuroscientists.  They didn’t like what they saw.

Ogi attempts to engage with respondents

The lead researcher opened a journal (now showing a single entry, an apology) for the purposes of answering questions about the research; and in the space of about two days, that journal moved from polite, rather subservient requests for clarification, to a full-on flamewar, as the lead researcher put up his questions for comment. As he engaged, he revealed more and more of his (very strange) thinking (he’s deleted his comments on this thread, but you can work some of them out), and his subjects began to research him in earnest.

Google is your friend (and Wikipedia, and Youtube)

Turns out, Ogi Ogas had forgotten to mention a few things:

  1. He wasn’t actually affiliated with Boston University any more
  2. While they were indeed neuroscientists, their Ph.Ds were on visual processing and artificial intelligence
  3. The lead researcher’s Ph.D was funded by the US Department of Homeland Security
  4. The lead author gained earlier infamy as a successful contestant on the American version of ‘Who Wants To Be a Millionaire’

And, last but not least, there was another teeny fact missing:

The authors had just signed a substantial book deal with Penguin for a popular science book entitled: ‘Rule 34: What Netporn teaches us about the brain.’

(As one commenter put it: ‘What? You think we can’t Google?’)

(NB – the literary agency has changed the book title now, to ‘Rule 34′)

So they asked about these Netporn theories, and then the shit really hit the fan.  It’s hard to follow the logic, but his theory (screencapped here)  drew on data-mining of adult sites aimed at men, and posited that explicit fanfiction for women could be equated with male interest in male-to-female transsexuals  (?!) and that both of these things could be used to model subcortical processing (whatever that is) in male and female  brains.  Or something.

Somewhere around there, people stopped arguing with him and started taking direct action.  The academics started complaining to Boston University, the creatives started creating cat macros, the neuroscientists started writing long introductions to neuroscience and the specialists in gender identity just started screaming.  There were a few more updates, and then Ogi locked his journal.  He issued a few wandering emails, and removed most of his journal (and indeed many of the comments that he’d left elsewhere).  Naturally, the LJers (being used to the ways of flamewars) took screenshots of the more alarming content well in advance.

Aftermath

From beginning to end, Ogi Ogas maintained that he wasn’t doing social research, he was just collecting data.

The day after the shitstorm, someone reported their conversation with the University of Boston’s rearch ethics board: he wasn’t formally affiliated, and he didn’t have ethics board clearance.  His university pages have now disappeared, the questionnaire is down, and at time of writing, he seems to be deleting all his comments elsewhere.

On the face of it, this is simply an extreme example of shoddy and unethical  research which will reflect badly on anyone who tries to do research online, especially within a community or subculture.   Anyone who approaches that particular community in the future is going to encounter deep suspicion.

It goes further, though.  One of the very odd features of the whole story is that Ogi Ogas and his colleague took a lot of care to approach prominent people. He got a great deal of help from some of them (he also got a magnificent brush-off from one, but that’s another story*).   All of those people are writing to explain that he seemed genuine, and they trusted him.   They offered the same critique of the questions that anyone would.  He seemed to listen, but went ahead with his own version.  This is either arrogance or sociopathy.

One of the people he approached has written to apologise for being taken in, and to reprint some of their correspondence.  She warns him that his attempts to research this particular community are probably dead in the water.   In his reply to her, he’s chirpy.

‘Eventually we’re going to go through this all over again with the far right. It will be interesting to see who throws the meaner punch.’

And I’m left thinking: is this the ultimate troll?

The book is due out in 2010.

*My favourite part of these people’s very lengthy smackdown is the grand postmodern refusal:

‘And so we decline to be interviewed by you; we decline to be the objects of your fascination; we decline to be naturalized; we decline to allow our political project to be cited in support of the very discourses we are trying to question.’

ETA: When respondents bite back

I actually hesitated in writing this up, because I was worried that mainstream researchers will see this as a distant kerfuffle in an unlikely subculture.   But I agree strongly with the writer at the Rough Theory blog (see below), who suggests that Ogas may fail to take valid community criticisms seriously, because he has so thoroughly Othered them as respondents.  In other words, ‘they’re so weird, we don’t have to be careful with them.’

The second general learning point for anyone thinking of attempting a controversial online questionnaire, is how quickly things go viral.  Ogas was terribly happy about the response rate (reliability and validity were not a concern); that same speed of process led, very rapidly, to critique, opprobrium, and direct action.   Before you engage?  Do us all a favour and go on that Methodology course.

Some other quick links:

Rough Theory’s roundup

Unfunny Business’s summary of the whole mess

Feminist SF

Jonquil’s thoughts on respondents who bite back

{ 25 comments }