Lies, Darned lies

I’ve blogged before about the fact that we know so much more about the way people engage in culture than we did 15 years ago.  There’s an irony that some frankly stupid top-down targets have led to the collection and analysis of data in a sophisticated and highly useful way.  For example, in England the Taking Part Survey was established to measure some very blunt and technically unachievable targets.  But the data gathered from that survey, melded with other datasets, allowed Arts Council England to produce their brilliant segmentation modelling that has proved extremely popular and practical with arts organisations.

With the abandonment of targets at a local and national level, there is no need to invest in collecting data to measure performance in the same way.  I do worry that with the cuts ahead surveys like Taking Part could therefore prove an easy target.  If they go, we lose much of our ability to keep on top of emerging trends and therefore fund and deliver the cultural opportunities people want.

For good (financial) reasons this year’s Taking Part Survey in England had a far smaller sample.  There’s nothing wrong with this in itself, but the margins of error increase, particularly when you are trying to delve deeper into the data.  Journalists don’t seem to understand this.  I am not a statistician, but even I could see that there was an awful lot of (statistically speaking) rubbish being printed when the figures were released last month. Did the proportion of the adult population visiting a library continue to fall?  Technically no.  But if the sample size had been as big as previous years we could be a lot more confident about the figures. Did this get explained clearly anywhere in the media?  Not that I’ve spotted.

And then there was the Threadneedle Prize Poll.  Hats off to them: what a wonderful PR job in getting coverage for their event on Monday evening.  But was this a robust piece of research presented in a balanced way in its Press Release?  Of course not.  A good dissection of some of the issues about how the questions were asked is available here. But it was swallowed whole and regurgitated by the media.  Even the BBC coverage lacked any challenge or balance.

So two recommendations:

1.       If there have to be savings on (robust) national surveys, let’s do them less often rather than in less detail.

2.       All journalists should have to go on a basic course in understanding statistics before they are allowed to quote any figures in an article.

David Brownlee, Chief Executive, Audiences UK