Sunday, September 14, 2008

Patroclus Reviews A Book, Rants

I'd almost finished Ben Goldacre's new book 'Bad Science' when my cousin popped up to divulge some salacious details of a date she once had with him, which was fortunate, because if I'd known about this earlier my judgment of his oeuvre might have been clouded*.

Goldacre: 'fabulous hair', says source.

But happily ignorant of a number of intriguing snippets of information regarding the good doctor, I was able to enjoy the first 292 pages on an entirely intellectual, rather than visceral, level. Which is good, because Ben Goldacre's entire professional raison d'être is to teach us to use our brains, not our emotions, to assess the science and health stories we read in the media.

I realise I'm probably preaching to the converted here, what with the vast majority of this blog's readers being Guardian-reading (and even Guardian-writing) types who are well aware of Dr Ben's Bad Science column and his apparently single-minded dedication to debunking all those lurid and ill-researched newspaper stories about health scares, miracle cures, fad diets and all the rest.

Anyone who saw yesterday's Guardian front-page story, for example, will know that Dr Ben isn't afraid to explain exactly why so many 'leading' nutritionists and vitamin-supplement advocates are quacks, frauds and charlatans of the worst sort, even if doing so lands him and the Guardian in court.

His book, then, is all about how lifestyle and consumer journalists lack the scientific knowledge and analytical skills to 'see through' pseudo-scientific claims about health risks - such as the supposed 'link' between the MMR jab and autism - with the result that newspapers are full of badly reported, badly researched and poorly-backed-up scare stories that appeal to readers' emotions rather than to their reason.

On a general level, the book not only provides the reader with a battery of critical tools with which to deconstruct and interrogate these so-called news stories - and learning to read between the lines of media stories, as far as I'm concerned, is an essential life skill and one I'll be teaching the Blue Kitten the minute she demonstrates any kind of mastery of the art of reading - but it's also very, very scathing and frequently very, very funny.

For me personally, having worked for several years in PR, I also found a lot of it uncomfortably close to home. Because, as Dr Ben rightly points out, the reason these scare stories end up in the media in the first place is that there's a PR machine behind them, creating sensationalist press releases from 'research' that is at best deeply flawed and at worst completely made up. The aim of this PR machine, usually, is to promote some vitamin supplement, specialist diet or homeopathic remedy as being better than anything suggested by mainstream medicine, the pharmaceutical industry or your own common sense.

Thankfully I've never done PR for any product that actually had a direct effect on life and death - unlike the vitamin supplements promoted by Matthias Rath as being more effective against Aids than anti-retroviral drugs - but I'm uncomfortably familiar with the process by which a press release purporting to be about a piece of research ends up being widely reported by gullible, inexperienced or just plain busy journalists.

I've lost count of the number of press releases I've seen that announce 'important research findings' without mentioning what the research consisted of, how it was conducted, how many people were studied, or who commissioned it whether it was independent or commissioned. And usually, there's no actual research report for journalists and interested parties to peruse, only the press release itself.

This is because very often the research has been conducted by a PR person emailing a bunch of their friends with a poorly-designed survey, and reporting the results as percentages rather than absolute numbers. '65% of Britons have been victims of identity theft' sounds like a good story. But if you only survey 14 people - and only people that you know - then the fact that nine of them have suffered identity theft means nothing. You might have consciously selected people you know to have been victims, for example. Or your friends might consist predominantly of people who spend a lot of time divulging personal information on Facebook, and are therefore pre-disposed to having their personal details stolen.

What amazes me more than the utter lack of any kind of intellectual rigour involved in this PR wankery, though, is the willingness of journalists to report it as bona fide fact. I know that editorial staffs are forever being cut down, and that most journalists don't have time to investigate every story properly. But I don't think there's any excuse for uncritically publishing meaningless statistics as if they were hard evidence of some supposed trend**. It only encourages PR people to put even less effort into their so-called surveys, resulting in an ever-diminishing respect for factual accuracy and an entire newspaper-reading public who think they're being informed, but are actually being fed a diet of made-up rubbish dressed up as fact.

Which, when it's made-up rubbish about identity theft, may not be so bad, but when newspapers report that MMR causes autism, or that vitamin C can reverse the spread of Aids, is not only irresponsible but actively evil.

This post was brought to you by raspberry leaf tea, chocolate Hobnobs, and the continued non-appearance of the Blue Kitten.

* Naturally we're far too classy for kiss and tell on this blog, but I will just say that the phrase 'a bit of public frottage on Greek Street' remains indelibly etched in my mind.

** I read the other day, for example, that according to the UK Equality and Human Rights Commission, only 13.6% of national newspaper editors are female, compared with 17.4% a year ago. Along with many other media outlets, the Guardian - Ben Goldacre's own paper - interprets these figures as illustrative of how the number of women in senior management roles is receding.

Now I'm prepared to believe that the EHRC has ample quantitative evidence for the return of the glass ceiling, but this claim in particular is statistically invalid and can't be used as evidence of anything. There are only around 25 national newspapers in the UK. This means that the EHRC's stats show that there's one fewer female editor now than there was a year ago. When your so-called stats are dictated by the actions of one single person, they aren't representative of a national trend, sorry EHRC.


ScroobiousScrivener said...

Hm. In general of course I completely agree with you, but on the newspaper editor thing, I rather want to see (and I don't find in your link) exactly what they mean by "newspaper editor". Because there are an awful lot of job titles on newspapers that include the word editor (I mean managing editors, section editors, deputy editors etc, rather than sub-editors or production editors) and if they were including all those, it would be a rather more valid point. I admit it's *probably* more likely that they mean simply The Boss, but it's in my nature to immediately think "yesbut..." Sorry.

patroclus said...

This is a very good point Scroobious - their report (download PDF) doesn't specify what they mean by editor, nor how many newspapers they counted. It just says they compiled the info from visiting national newspaper websites. The figure is also missing from the graphic on p10, which makes me even more suspicious. But I'm thinking that someone who knows maths could probably work out from the percentages themselves how many individuals are involved.

Incidentally, the number of female editors may be down from last year, but it's up from 2003 and 2004, when 9.1% of national newspaper editors were apparently female, and from 2005, when 13% were. So not quite so much of a reversal of fortune as we're led to believe.

This isn't at all to say that I don't believe there are fewer women in power these days - I'm just saying that using percentages with what seems to be such a small sample is misleading.

Annie said...

I like your hi-tech new meatspace blogroll, very flash. I wish I knew you IRL so I could get on it.

Geoff said...

Bad science, good hair.

A little tense around the shoulders, though. He needs to see a good reflexologist.

Valerie said...

One of the best classes I took in graduate school was a class in assessing and creating good experimental technique. Along the way, we learned how to read a research paper and just how much weight to give the experimental work therein. Almost all published research has some flaws, but some studies are better than others.

When I am motivated, and I read some claim in SELF or some other popular rag about health or ecology or some such, I actually look the dem' thing up on MEDLINE or wherever and read their methodology. And about, oh, 4 times out of 5 (note: not a scientifically derived number) I determine the claim is baloney -- or at least, not yet supported properly by research.

Also? I am going to see Nick Cave Tuesday night. You may be jealous now.

Good luck with the tea and Hobnobs approach. (I'd put more faith in the Hobnobs at this point. Mm, biscuits.)

rivergirlie said...

one unfortunate side effect of the fall of the berlin wall is that journalists can no longer blithely refer to 'recent research in the USSR', confident in the knowledge that no-one will ever be able to check. it was terribly when you had no evidence whatsoever to support what you were saying. not that i would know anything at all about such behviour ...

rivergirlie said...

useful - terribly useful. that's what i meant. x

patroclus said...

Annie: It's good isn't it - I'll get round to doing the other blogroll as well, but I have to input all the URLs by hand, and [continues whingeing in this vein for several minutes...]

Geoff: I stole that picture off the Daily Mail, so they were probably trying to make him look a) scared and b) untrustworthy.

Valerie: Yes, if there's one thing I've learned from this book it is that all research is flawed, but to different degrees, and the best research is research that admits to its own shortcomings. Enjoy Nick Cave, he always puts on a good show. Is it one of his solo/Grinderman gigs, or is it with the Bad Seeds?

RG: Why not just substitute with 'Recent research in North Korea...'?

jill said...

Incidentally, the number of female editors may be down from last year, but it's up from 2003 and 2004, when 9.1% of national newspaper editors were apparently female, and from 2005, when 13% were. So not quite so much of a reversal of fortune as we're led to believe.

This isn't at all to say that I don't believe there are fewer women in power these days - I'm just saying that using percentages with what seems to be such a small sample is misleading.

Yes - see also, "Global warming? Can't really exist - it was really chilly last week." I almost have to consciously part my jaws so as not to grind my teeth at this sort of idiocy.

And on your follow up re: That Woman on your last post (she who will not be named by me because she makes me want to a. vomit, b. move to a swing state, just so I can add one more non-That-Woman-and-The-Old-Guy vote to their rolls, or c. kill myself because my countrymen are falling into the worst possible anti-cranial, anti-sense, anti-evidence-based logical fallacies in order to throw our national politics down yet another raving shithole) I somehow feel moved to apologize for my country again. I hate that our politics make me constantly want to do that (especially for the last eight years).

Also - the captcha for this comment is "cotot" - which can be parsed as "cot" or "tot," which I take to be a promise of Chaton Bleu's imminent arrival. Here's hoping.

jill said...

Also: Hobnobs? Plain or milk?

oyebilly said...

If I was going to engage in public frottage, Greek street wouldn't be my choice.

patroclus said...

Jill: Cheer up, they haven't won yet. Also, we somehow voted comedy right-wing mophead Boris Johnson to be mayor of London, and it looks like we're going to be voting in a Conservative government at our next election, so we've no leg to stand on, really. Re. Hobnobs, ideally plain, but Asda (local supermarket, colonial outpost of Wal-Mart) only stocks milk ones, so they have to suffice.

Billy: Out of interest (and in the spirit of scientific enquiry), which street would you choose?

Tim Footman said...

...whereas, let us say, a discreet handjob in Soho Square is perfectly acceptable?

Anonymous said...

on this have you read flat earth news by nick davies? It's not bad: basically he tries to explain why we get such rubbish news all the time, eg., MMR scare, 2YK, etc.

In essence he argues we are caught between the PR industry, a severely malnourished journalist pool who have simply no time to check their sources, an emphasis on immediate news, the echo chamber effect and corporate profit--the end result is that 'the news' we get is often recycled, of no substance, rarely doubled checked for accuracy or bias.

have to say after reading it, I had a fair amount of sympathy for everyday journos who are hardly the stuff of 'the state of play' but more like sad drones who spend most of their day in a cubile cutting, pasting, paraphrasing articles from AP, reuters and PR copy in order to meet some quota of copy. I certainly felt, after reading it, that it's just too easy to blame journalists and editors. Davies suggests the problem is far more structural than individual.

Anonymous said...

gah that's 'cubicle'...

hope the blue kitty birth goes well....

Tim Footman said...

Charlie Booker touches on similar statistical idiocy here (the bit about men and their pubes).

But I suspect your mind may be elsewhere right now.

Boz said...

And his mission with health stories is alas true of a lot of other sectors of news reporting.

And this bothers me enormously, because I work in PR. Quite often we haggle on the angle we take with a press release with a client, but we would never issue anything a client wasn't happy with or that we didn't feel was accurate - because we're lucky enough to work in areas so transparent that any such ball cocks would quickly get spotted.

So, in short, nothing to do with research would ever cross my desk that did not fully explain how the research had been done.

But yes, as Tim points out, your thoughts will be elsewhere...

(PS - I don't work in healthcare PR)

Anonymous said...

It seems almost wrong to be commenting on this when the blue kitten is so much more significant and no doubt the focus of all your attention for some time to come, but I thought you might be interested in this on the failings of surveys.