The Spirit Level has been debunked. More or less.

The BBC Radio 4 show More or Less looks at the facts behind well publicised statistics. The show is presented by the author of The Undercover Economist, Tim Harford and, as a regular listener, I was intrigued to hear that the first show of the new series promised to 'decode The Spirit Level debate'. For a show dedicated to debunking junk statistics it was obvious subject matter, but I wondered how Harford could 'decode' such a voluminous topic over the airwaves in one show.

In the end, he didn't need to. From the outset, Harford admitted there were too many competing claims to fit into a magazine show, and instead interviewed The Spirit Level's co-author Kate Pickett, who did more damage to the reputation of The Spirit Level in the space of ten minutes than any number of supposed "idea wreckers".

The interview (listen here for the next couple of days, or subscribe to the podcast) is essential listening if you've been following the controversy, as Pickett struggles to answer some fair and simple questions. Strangely enough, Wilkinson and Pickett's Equality Trust website—which is normally so quick to let people know when The Spirit Level has been mentioned in the media—have yet to post a link to this interview. And since the audio file won't be available for much longer, I've transcribed some of the key moments for posterity...


Falling back on other people's research

Wilkinson and Pickett's first line of defence is to claim that there are 100s of peer-reviewed studies which support their conclusion. As I have said before, this is just not true. Most of the studies they reference in The Spirit Level do not even mention income inequality.

In the More or Less interview, Kate Pickett once again claimed there was a "vast body of research" behind The Spirit Level. Tim Harford picked her up on it...

KP: We wrote a book that's intended to be a synthesis of a very vast body of research. Not only our own, but those of other people... There is a consistent and robust and large body of evidence showing the same relationship.

TH: That's an interesting point that you make. Often, in response to critics, you have referred not to your own book, not to your own data but to other published research. I'd really like to focus on the research that's presented in your book. It's very easy to say there are 50 papers, there are 200 papers, that support our research but we don't really know how you've selected those papers.

KP: We actually have completed a systematic review of all of the studies of income inequality and health, and we reference that in our book. We do examine things systematically and certainly—when we are doing our own research, publishing in peer-reviewed journals—we have to be aware of all the literature in the field. But that doesn't mean that every paper in the field has good methods, comes to the right conclusion, studies the right thing.

TH: I absolutely agree. One of the papers that you refer to in support of your argument on women's empowerment and women's status which was published in 1999 by Kawachi and some other authors, you claim supports your findings on women's status and income inequality. I've looked at their abstract. It doesn't seem to attack that question at all. It's simply on another subject—a somewhat related subject but not on the subject of income inequality.

KP: They've definitely published and we may have inadvertently put the wrong reference into that document [laughing nervously]. But Kawachi and Kennedy have certainly published finding a relationship between income inequality and women's status. The paper is Women's Status and the Health of Women and Men: a view from the States and it was published in Social Science and Medicine in 1999.

TH: That's the one I'm looking at.

Note: On page 58 of The Spirit Level, it states: "Researchers at Harvard University showed that women's status was linked to state-level income inequality. (36)"

Reference 36 is the Kawachi study ('Women's status and the health of women and men: a view from the States', 1999). As its title suggests, this study compared women's status with health, not with inequality. Indeed, the authors found a correlation between women's status and health even after controlling for income inequality.

Failure to look at other variables

The Spirit Level relies on the conceit that countries are fundamentally the same, with income inequality being the main variable that distinguishes them. This allows Wilkinson and Pickett to disregard other variables such as income, culture, history, demography, ethnicity, geography, law, politics and climate. Ignoring other variables and confounding factors would be a flaw in any study—as Harford points out, it breaks a basic rule of epidemiology—but when entire countries are being studied, this flaw becomes overwhelming. Pickett's response is revealing: she and Wilkinson do not "believe" that factors other than income inequality have an effect on a country's performance, so they don't bother looking at them.

TH: All of your studies are what are called bivariate analysis. In other words, they're all income inequality plotted against some other variable. Now, my understanding of best practice in social sciences is that you would always control for other variables. You would include 2, 3, 4, 5, 6 other variables and...

KP: Well, you wouldn't do that arbitrarily. You would do that if you believed those variables were potential alternative explanations of the relationship you're looking at.

TH: So, if I understand your statement correctly, you didn't include any multiple variable analysis because you just think that actually none of these variables are of interest—none of them are potential alternative explanations and you can just do the simple income inequality versus x analyses?

KP: That's right, but of course, again, other researchers have conducted studies that do control for more, where, as well as examining the effect of income inequality at the level of the whole society, people include individual's own levels of income or levels of education in those analyses and, again, those bear out our findings in relation to health.

TH: We come again to...you're basically rowing back from your analysis and saying...

KP: No. Indeed I'm not...

TH: "Don't look at our analysis, look at these other people because they support us."

KP: We believe that to control for individual income is actually over-controlling, so we would not consider that best practice.

Academic criticism

Although well received by some journalists and politicians, The Spirit Level has received a much cooler reaction from academics. One of the few serious academics to have reviewed the book was John Kay, Professor of Economics at London Business School and former Director of Institute of Fiscal Studies. Pickett's response to Kay's review speaks volumes...

TH: When John Kay reviewed your book in the Financial Times —and I believe John Kay would be broadly sympathetic to your idea that egalitarianism is important—he wrote: "The evidence presented in the book is mostly a series of scatter diagrams with a regression line drawn through them. No data is provided on the estimated equations, or on relevant statistical tests. If you remove the bold lines from the diagram, the pattern of points mostly looks random, and the data dominated by a few outliers." Do you think that's fair?

KP: No, I don't think it's fair. [Testily] He didn't read the book thoroughly, obviously.

Outlandish claims 

In the last chapter of The Spirit Level, Wilkinson and Pickett make some extraordinary claims about what could happen if Britain reduced income inequality to Scandinavian levels. These include: teen births falling to a third of current rates, mental illness being halved, life expectancy rising by a year and the murder rate falling by three-quarters. Harford asks her about the last of these predictions.

TH: Clearly your book is a systematical analysis and partly also a political book. You have a political case to make—there's nothing wrong with that. You have public policy actions that you would like to see taken. But do you think you may have overstated some of those? Let me give you an example. On page 268 of you book—towards the conclusion—you say that if Britain became as equal as Japan, Norway, Sweden and Finland, homicide rates could fall by 75%. But as I'm sure you've had pointed out to you by now, the UK's homicide rate is already below the average of those four countries.

KP: It's not actually. It's been pointed out that it's below Sweden. It's not below the average of those countries. Those claims [ie. in The Spirit Level], they're based on regression models and of course they're only as good as they're model they're based on.

TH: [Incredulously] But.. sorry... but you've made that claim!

KP: Yes, yes, we do...

TH: And you stand by it?

KP: Yes. That Britain would become a much healthier and more socially better functioning place if it were more equal.

TH: You said that if Britain became as equal as these four countries, homicide rates could fall by 75%. Do you not feel that's really overstating the case, or do you stand by that?

KP: That's based on the model. I mean, I think we could try it and see.

[end of interview]

TH: Kate Pickett, co-author of The Spirit Level. We did go to her Equality Trust website, by the way, and downloaded the data on homicide rates in the UK and in the relevant four countries and it does seem that I was right to say that the UK's homicide rate is already below the average of those four countries. You're listening to More or Less...