YouGov’s Great Wall of Secrecy, Bluster and Evasion

Thursday, 03 July 2014 16:36

25 Comments

  By James Kelly
 
YouGov have been on the warpath over the last couple of days.  Nothing unusual in that, you might think – all of the polling firms that have been active in the referendum campaign so far have naturally been keen to defend their reputations, their methodologies and their results against any criticism. 
 
However, most have been circumspect enough to point out that they are simply doing their level best in very challenging circumstances, and that it’s impossible to be 100% certain they’re getting it right, given that there have been no previous independence referendums in which rival methodologies could be put to the test.

Such a cautious and diplomatic approach hasn’t proved appealing for Peter Kellner, the President of YouGov and a man with strong Labour connections, who has instead penned an extraordinary blog on his firm’s website that explicitly attacks the methodology of the three polling companies that are currently showing a very close race in the referendum.  His conclusion is breathtaking in its starkness – YouGov are definitely right, the Yes-friendly pollsters are definitely wrong, and this means that Scotland is guaranteed to reject independence in September by a convincing margin. 

The leap of “logic” required to take him to the latter quasi-religious declaration of certainty is nothing short of astounding.  In reality, even if YouGov do turn out to be the most accurate firm (and remember they’re currently the most No-friendly pollster of the lot), it categorically does not mean that the Yes campaign have already lost – it simply means that a more substantial swing will be required over the closing months of the campaign.

In spite of the technical detail Kellner goes into in support of his attack on fellow pollsters, my view is that this is a monumental bluff on his part.  For more than three years, he has been telling anyone who will listen that it is literally impossible for Yes to win the referendum. 

Indeed, at one point, he went so far as to suggest that the SNP government knew perfectly well that a straight Yes/No referendum on independence was unwinnable, and that the proposal to hold one was simply a tactic to secure a multi-option vote including the option of Devo Max.  Having been comprehensively proved wrong on that point, it’s now even more crucial to his reputation as a polling analyst that he isn’t similarly proved wrong on his claim that Yes don’t stand a chance, and have never stood a chance. 

The existence of three polling firms that currently show Yes within a hair’s breadth of victory is plainly a huge threat to the Kellner narrative, and his blogpost appears to be an attempt to bully (or perhaps ‘spook’ is a better word) those firms into adopting a more No-friendly methodology.  At least that way there would be safety in numbers for Kellner and YouGov, with all of the pollsters standing or falling together. 

It’s a cynical ploy that deserves to fail, and it must fail.  With the future of this country at stake, it would be an utter disgrace if pollsters suddenly copped out and started playing ‘follow the leader’ – particularly when that leader has such a transparent agenda.

It’s unsurprising that Kellner chooses to concentrate his rather patronising fire on the relatively new pollster Survation.  He’d have much more difficulty in being taken seriously if he directed his onslaught at ICM, who are also one of the more Yes-friendly pollsters at present.  ICM certainly haven’t been immune from criticism in this campaign – they used a leading question in one notorious poll, and have produced much more volatile results than other firms. 

But at the end of the day they’re a more experienced firm than even YouGov, and are often billed as the UK’s “gold standard” pollster.  The idea that such a seasoned and professional organisation won’t have taken full account of the technical issues raised by Kellner is risible – and yet they’ve still reached a considered view that the most appropriate methodology is one that usually (not always) produces a significantly higher Yes vote than YouGov.

Even before the Kellner blogpost appeared, I had my own little taste of YouGov’s new belligerent tactics.  Shortly after the firm’s latest referendum poll was published on Monday evening, Kellner’s colleague Laurence Janta-Lipinski was on Twitter bemoaning the fact that so many people are beastly about YouGov, rather than simply accepting the obvious truth that they’re more accurate than other pollsters. 

He seemed to be casting around for a critic of the firm to ‘make an example of’, and to my surprise I turned out to be the chosen ‘victim’.  I had just written a blogpost about the YouGov poll, in which I listed the standard reasons for at least maintaining a healthy scepticism about the firm’s methodology. 

Curiously, though, it was one of the more innocuous points I made that attracted Janta-Lipinski’s ire, namely my suggestion that YouGov are more secretive than other pollsters.  That’s simply an indisputable fact – other pollsters provide raw numbers in their datasets, whereas for some reason YouGov only provide percentages.  That’s an irritant to polling geeks like myself, because it means we can’t calculate the voting intention numbers down to one or two decimal places.  But it’s scarcely the most important of complaints.

Nevertheless, Janta-Lipinski feigned astonishment that anyone could ever dream of accusing YouGov – YouGov of all firms! – of a lack of transparency, and demanded to know what I was getting at.  Quite honestly, it’s so unusual for a YouGov representative to pay any attention to riff-raff like myself that I didn’t want to waste a golden opportunity by getting into a pointless discussion about the use of percentages in datasets, so instead I placed more emphasis on an example of YouGov’s secretiveness that I consider to be much more significant.  Janta-Lipinski’s reaction was dismaying and fascinating in equal measure.

Like all online polling firms, YouGov weight their sample by recalled vote from 2011.  This is to ensure that the sample is politically representative, and doesn’t include too many Tory voters, or too few SNP voters, or whatever.  But YouGov take an additional and rather eccentric step – they split SNP voters, and SNP voters only, into two distinct groups and then weight them separately. 

This seems to be the most plausible explanation for why YouGov routinely produce much more No-friendly headline numbers than other online firms.  One of the two SNP groups is always upweighted dramatically, while the other is not. 

The likelihood is that YouGov are artificially giving far more weight to the type of SNP voter who is likely to vote No – but we don’t know that for sure, because the firm have persistently kept the voting intention figures for the two groups under wraps.  Instead, all we see in the datasets are the combined voting intentions of the two SNP groups after the weighting has been applied, which tells us nothing at all.

So I essentially laid down a challenge for Janta-Lipinski – if he truly believes that YouGov have no issues with a lack of transparency, would he undertake to publish the voting intention breakdown for the two SNP groups, and if not, why not?  Bearing in mind that he had just demanded an explanation from me, it didn’t seem unreasonable or unfair to raise a query with him. 

But suddenly he clammed up, and started working his way through a repertoire of flimsy excuses for why he couldn’t possibly answer the question right away.  After I pressed the point further, he eventually haughtily informed me that the only reason we wouldn’t be given the information is quite simply that we aren’t entitled to it – YouGov don’t provide “thousands of potential xtabs” at the “behest of bloggers”.  And this was from a man who couldn’t conceive of YouGov ever being accused of secrecy!

At that point I knew I was dealing with people who were withholding information for a reason they didn’t want to admit to.  Janta-Lipinski’s implication that this is merely one trivial detail out of “thousands” is plainly risible, especially in the light of Kellner’s blog, which specifically cites the splitting of SNP voters into two groups as a primary reason for thinking that YouGov are getting it right and others are getting it wrong. 

So what is the real explanation for the obsessive secrecy?  My suspicion is that YouGov know that if they showed the huge disparity between the voting intentions of the two groups, the strangeness and artificiality of what they’re doing would start to hit home, and questions would be raised about whether the small ‘No-friendly’ SNP group that is always so sharply upweighted is really typical of the huge section of the population that it is supposed to represent. 

The more a sample has to be upweighted, the greater the potential for error.  There’s also the question of why such a substantial upweighting is always required – couldn’t that be an indication that the basic assumptions being made are misconceived? 

None of this necessarily means that YouGov’s headline numbers are wrong, but it does mean they’re walking a tightrope and privately keeping their fingers very firmly crossed that it will be all right on the night.  Pretty much like all the other pollsters that Kellner looks down his nose at, in fact.

Janta-Lipinski did go on to indicate to me that he might eventually answer the questions he had evaded, but only after Kellner’s blogpost had appeared.  So I left a comment on Kellner’s post asking three specific questions –

1) Why the obsessive secrecy about the voting intentions of the two SNP groups?

2) If there is logic in splitting SNP voters into two groups and weighting them separately, why aren’t other potentially distinctive groups also separated out?  Doesn’t the current inconsistent approach smack of an organisation that is working backwards to generate the headline numbers that “feel right”?  (Something that Kellner is scarcely capable of being objective about, given his absurd declaration three years ago that the referendum was unwinnable for Yes.)

3) Given that we know a previous YouGov referendum poll was way out of line with the 2011 census in containing far too many English-born people and too few Scottish-born people, what steps have been taken to correct for this bias which is bound to skew results towards No?  If no corrective steps have been taken, why not?

I regret to inform you that none of these questions have been answered as of yet.  YouGov’s Great Wall of Secrecy, Bluster and Evasion remains unbreached.

Read more from James Kelly at http://scotgoespop.blogspot.co.uk/