Monday 30 August 2010

The Spirit Level has been debunked. More or less.

The BBC Radio 4 show More or Less looks at the facts behind well publicised statistics. The show is presented by the author of The Undercover Economist, Tim Harford and, as a regular listener, I was intrigued to hear that the first show of the new series promised to 'decode The Spirit Level debate'. For a show dedicated to debunking junk statistics it was obvious subject matter, but I wondered how Harford could 'decode' such a voluminous topic over the airwaves in one show.

In the end, he didn't need to. From the outset, Harford admitted there were too many competing claims to fit into a magazine show, and instead interviewed The Spirit Level's co-author Kate Pickett, who did more damage to the reputation of The Spirit Level in the space of ten minutes than any number of supposed "idea wreckers".

The interview (listen here for the next couple of days, or subscribe to the podcast) is essential listening if you've been following the controversy, as Pickett struggles to answer some fair and simple questions. Strangely enough, Wilkinson and Pickett's Equality Trust website—which is normally so quick to let people know when The Spirit Level has been mentioned in the media—have yet to post a link to this interview. And since the audio file won't be available for much longer, I've transcribed some of the key moments for posterity...

Falling back on other people's research

Wilkinson and Pickett's first line of defence is to claim that there are 100s of peer-reviewed studies which support their conclusion. As I have said before, this is just not true. Most of the studies they reference in The Spirit Level do not even mention income inequality.

In the More or Less interview, Kate Pickett once again claimed there was a "vast body of research" behind The Spirit Level. Tim Harford picked her up on it...

KP: We wrote a book that's intended to be a synthesis of a very vast body of research. Not only our own, but those of other people... There is a consistent and robust and large body of evidence showing the same relationship.

TH: That's an interesting point that you make. Often, in response to critics, you have referred not to your own book, not to your own data but to other published research. I'd really like to focus on the research that's presented in your book. It's very easy to say there are 50 papers, there are 200 papers, that support our research but we don't really know how you've selected those papers.

KP: We actually have completed a systematic review of all of the studies of income inequality and health, and we reference that in our book. We do examine things systematically and certainly—when we are doing our own research, publishing in peer-reviewed journals—we have to be aware of all the literature in the field. But that doesn't mean that every paper in the field has good methods, comes to the right conclusion, studies the right thing.

TH: I absolutely agree. One of the papers that you refer to in support of your argument on women's empowerment and women's status which was published in 1999 by Kawachi and some other authors, you claim supports your findings on women's status and income inequality. I've looked at their abstract. It doesn't seem to attack that question at all. It's simply on another subject—a somewhat related subject but not on the subject of income inequality.

KP: They've definitely published and we may have inadvertently put the wrong reference into that document [laughing nervously]. But Kawachi and Kennedy have certainly published finding a relationship between income inequality and women's status. The paper is Women's Status and the Health of Women and Men: a view from the States and it was published in Social Science and Medicine in 1999.

TH: That's the one I'm looking at.

Note: On page 58 of The Spirit Level, it states: "Researchers at Harvard University showed that women's status was linked to state-level income inequality. (36)"

Reference 36 is the Kawachi study ('Women's status and the health of women and men: a view from the States', 1999). As its title suggests, this study compared women's status with health, not with inequality. Indeed, the authors found a correlation between women's status and health even after controlling for income inequality.

Failure to look at other variables

The Spirit Level relies on the conceit that countries are fundamentally the same, with income inequality being the main variable that distinguishes them. This allows Wilkinson and Pickett to disregard other variables such as income, culture, history, demography, ethnicity, geography, law, politics and climate. Ignoring other variables and confounding factors would be a flaw in any study—as Harford points out, it breaks a basic rule of epidemiology—but when entire countries are being studied, this flaw becomes overwhelming. Pickett's response is revealing: she and Wilkinson do not "believe" that factors other than income inequality have an effect on a country's performance, so they don't bother looking at them.

TH: All of your studies are what are called bivariate analysis. In other words, they're all income inequality plotted against some other variable. Now, my understanding of best practice in social sciences is that you would always control for other variables. You would include 2, 3, 4, 5, 6 other variables and...

KP: Well, you wouldn't do that arbitrarily. You would do that if you believed those variables were potential alternative explanations of the relationship you're looking at.

TH: So, if I understand your statement correctly, you didn't include any multiple variable analysis because you just think that actually none of these variables are of interest—none of them are potential alternative explanations and you can just do the simple income inequality versus x analyses?

KP: That's right, but of course, again, other researchers have conducted studies that do control for more, where, as well as examining the effect of income inequality at the level of the whole society, people include individual's own levels of income or levels of education in those analyses and, again, those bear out our findings in relation to health.

TH: We come again're basically rowing back from your analysis and saying...

KP: No. Indeed I'm not...

TH: "Don't look at our analysis, look at these other people because they support us."

KP: We believe that to control for individual income is actually over-controlling, so we would not consider that best practice.

Academic criticism

Although well received by some journalists and politicians, The Spirit Level has received a much cooler reaction from academics. One of the few serious academics to have reviewed the book was John Kay, Professor of Economics at London Business School and former Director of Institute of Fiscal Studies. Pickett's response to Kay's review speaks volumes...

TH: When John Kay reviewed your book in the Financial Times —and I believe John Kay would be broadly sympathetic to your idea that egalitarianism is important—he wrote: "The evidence presented in the book is mostly a series of scatter diagrams with a regression line drawn through them. No data is provided on the estimated equations, or on relevant statistical tests. If you remove the bold lines from the diagram, the pattern of points mostly looks random, and the data dominated by a few outliers." Do you think that's fair?

KP: No, I don't think it's fair. [Testily] He didn't read the book thoroughly, obviously.

Outlandish claims 

In the last chapter of The Spirit Level, Wilkinson and Pickett make some extraordinary claims about what could happen if Britain reduced income inequality to Scandinavian levels. These include: teen births falling to a third of current rates, mental illness being halved, life expectancy rising by a year and the murder rate falling by three-quarters. Harford asks her about the last of these predictions.

TH: Clearly your book is a systematical analysis and partly also a political book. You have a political case to make—there's nothing wrong with that. You have public policy actions that you would like to see taken. But do you think you may have overstated some of those? Let me give you an example. On page 268 of you book—towards the conclusion—you say that if Britain became as equal as Japan, Norway, Sweden and Finland, homicide rates could fall by 75%. But as I'm sure you've had pointed out to you by now, the UK's homicide rate is already below the average of those four countries.

KP: It's not actually. It's been pointed out that it's below Sweden. It's not below the average of those countries. Those claims [ie. in The Spirit Level], they're based on regression models and of course they're only as good as they're model they're based on.

TH: [Incredulously] But.. sorry... but you've made that claim!

KP: Yes, yes, we do...

TH: And you stand by it?

KP: Yes. That Britain would become a much healthier and more socially better functioning place if it were more equal.

TH: You said that if Britain became as equal as these four countries, homicide rates could fall by 75%. Do you not feel that's really overstating the case, or do you stand by that?

KP: That's based on the model. I mean, I think we could try it and see.

[end of interview]

TH: Kate Pickett, co-author of The Spirit Level. We did go to her Equality Trust website, by the way, and downloaded the data on homicide rates in the UK and in the relevant four countries and it does seem that I was right to say that the UK's homicide rate is already below the average of those four countries. You're listening to More or Less...


Unknown said...

Homicide stats from same source (UN)
UK (England+Wales) 1.2
SW 0.9
NO 0.6
FI 2.5
DK 1.2

The average of the 4 countries = 1.1, which is still below UK. If we were to include Scotland and Northern Ireland, the UK would be even higher. (Scotland 2.2, Northern Ireland 14.) So, how did TH get his numbers?

Christopher Snowdon said...

Wilkinson and Pickett used the average for 1990-2000 (as did Harford, presumably). Their figures can be downloaded from the Equality Trust website and they show (per million):

Norway: 9.70
Sweden: 18.70
Japan: 5.20

= Average: 15.45

UK: 15.00 per million

Anonymous said...

An important variable - almost always overlooked, is IQ.

Anonymous said...

Wealth inequality is also a problem in the U.S, a bit worse than the U.K.

Anonymous said...

Your calculation of the mean homicide rate is completely flawed. You are giving equal weight to countries of vastly different populations - Japan has a population of around 127 million whereas Finland's population is around 5 million. What you need to do is calculate the population weighted mean - this would give an average homicide rate of around 7 homicides per million, well below UK's rate of 15 per million.

Christopher Snowdon said...

No, that's not how it works. Countries are not weighted by population in The Spirit Level (quite rightly). Harford's calculation is quite correct.

Anonymous said...

We're not talking about individual countries here, we are talking about an average of a group of countries (Finland, Japan, Norway & Sweden). You are calculating an average of averages. By taking a simple mean of the four country's homicide rates you are giving equal weight to each of them, even though they are have very different populations. This matters a lot when their rates are so different (Finland's homicide rate is over five times that of Japan).

To calculate the average for the four countries you need to add up the number of homicides in each country and then divide by the sum of the populations.

Think of it this way - imagine you have two groups of people, Group A has 100 people in it and their average age is 70 years, Group B has 5 people in it and their average age is 10 years. So, what is the average age of the two groups? By your reckoning it should be (70 + 10)/2 = 40 years. Now, imagine you put the two groups together in the same room, would it look like the average age of those 105 people is 40 years? The true average age is (70 x 100 + 10 x 5)/(100 + 5) = 7050/1005 = 67.1 years.

Christopher Snowdon said...

I understood your point the first time. I should have explained more fully.

If the aim is to imagine that these four countries are a superstate with a single homicide rate, you're quite correct. The population of this superstate would be 90% Japanese and so naturally Japan's low homicide rate would drag down the rest.

That's not the issue Harford was looking at. He was testing the very specific claim that if the UK had a level of equality equal to Japan, Norway, Finland and Sweden, it would have a 75% lower homicide rate. To answer that, he did what The Spirit Level quite rightly does and gave each datapoint a value of 1. Population size does not affect either inequality or homicide so there is no reason to factor it in. In other words, the homicide rate in little Finland must be given equal weight to the homicide rate in big Japan.

And so, although a + b + c + d divided by 4 does not give you the homicide rate per million in an hypothetical superstate in which Scandinavians are a small ethnic minority, it does answer the question Harford was addressing of how one country compares to four countries.

Anonymous said...

Kawachi's study does mention women's status and inequlity, though it was only a minor point:

3.2.4. Relationships of women's status indicators to men's health, 1st paragaph:

"These correlations may partly reflect the fact that gender inequalities are manifestations of general inequalities. For example, the indices for female political participation and economic autonomy were both correlated (r=−0.49 and −0.36, respectively) with the Gini coefficient of income inequality (although neither the index of women's employment and earnings (r=−0.14) nor reproductive rights (r=−0.08) were correlated with the Gini index)"