Banner

  By James Kelly
 
YouGov have been on the warpath over the last couple of days.  Nothing unusual in that, you might think – all of the polling firms that have been active in the referendum campaign so far have naturally been keen to defend their reputations, their methodologies and their results against any criticism. 
 
However, most have been circumspect enough to point out that they are simply doing their level best in very challenging circumstances, and that it’s impossible to be 100% certain they’re getting it right, given that there have been no previous independence referendums in which rival methodologies could be put to the test.

Such a cautious and diplomatic approach hasn’t proved appealing for Peter Kellner, the President of YouGov and a man with strong Labour connections, who has instead penned an extraordinary blog on his firm’s website that explicitly attacks the methodology of the three polling companies that are currently showing a very close race in the referendum.  His conclusion is breathtaking in its starkness – YouGov are definitely right, the Yes-friendly pollsters are definitely wrong, and this means that Scotland is guaranteed to reject independence in September by a convincing margin. 

The leap of “logic” required to take him to the latter quasi-religious declaration of certainty is nothing short of astounding.  In reality, even if YouGov do turn out to be the most accurate firm (and remember they’re currently the most No-friendly pollster of the lot), it categorically does not mean that the Yes campaign have already lost – it simply means that a more substantial swing will be required over the closing months of the campaign.

In spite of the technical detail Kellner goes into in support of his attack on fellow pollsters, my view is that this is a monumental bluff on his part.  For more than three years, he has been telling anyone who will listen that it is literally impossible for Yes to win the referendum. 

Indeed, at one point, he went so far as to suggest that the SNP government knew perfectly well that a straight Yes/No referendum on independence was unwinnable, and that the proposal to hold one was simply a tactic to secure a multi-option vote including the option of Devo Max.  Having been comprehensively proved wrong on that point, it’s now even more crucial to his reputation as a polling analyst that he isn’t similarly proved wrong on his claim that Yes don’t stand a chance, and have never stood a chance. 

The existence of three polling firms that currently show Yes within a hair’s breadth of victory is plainly a huge threat to the Kellner narrative, and his blogpost appears to be an attempt to bully (or perhaps ‘spook’ is a better word) those firms into adopting a more No-friendly methodology.  At least that way there would be safety in numbers for Kellner and YouGov, with all of the pollsters standing or falling together. 

It’s a cynical ploy that deserves to fail, and it must fail.  With the future of this country at stake, it would be an utter disgrace if pollsters suddenly copped out and started playing ‘follow the leader’ – particularly when that leader has such a transparent agenda.

It’s unsurprising that Kellner chooses to concentrate his rather patronising fire on the relatively new pollster Survation.  He’d have much more difficulty in being taken seriously if he directed his onslaught at ICM, who are also one of the more Yes-friendly pollsters at present.  ICM certainly haven’t been immune from criticism in this campaign – they used a leading question in one notorious poll, and have produced much more volatile results than other firms. 

But at the end of the day they’re a more experienced firm than even YouGov, and are often billed as the UK’s “gold standard” pollster.  The idea that such a seasoned and professional organisation won’t have taken full account of the technical issues raised by Kellner is risible – and yet they’ve still reached a considered view that the most appropriate methodology is one that usually (not always) produces a significantly higher Yes vote than YouGov.

Even before the Kellner blogpost appeared, I had my own little taste of YouGov’s new belligerent tactics.  Shortly after the firm’s latest referendum poll was published on Monday evening, Kellner’s colleague Laurence Janta-Lipinski was on Twitter bemoaning the fact that so many people are beastly about YouGov, rather than simply accepting the obvious truth that they’re more accurate than other pollsters. 

He seemed to be casting around for a critic of the firm to ‘make an example of’, and to my surprise I turned out to be the chosen ‘victim’.  I had just written a blogpost about the YouGov poll, in which I listed the standard reasons for at least maintaining a healthy scepticism about the firm’s methodology. 

Curiously, though, it was one of the more innocuous points I made that attracted Janta-Lipinski’s ire, namely my suggestion that YouGov are more secretive than other pollsters.  That’s simply an indisputable fact – other pollsters provide raw numbers in their datasets, whereas for some reason YouGov only provide percentages.  That’s an irritant to polling geeks like myself, because it means we can’t calculate the voting intention numbers down to one or two decimal places.  But it’s scarcely the most important of complaints.

Nevertheless, Janta-Lipinski feigned astonishment that anyone could ever dream of accusing YouGov – YouGov of all firms! – of a lack of transparency, and demanded to know what I was getting at.  Quite honestly, it’s so unusual for a YouGov representative to pay any attention to riff-raff like myself that I didn’t want to waste a golden opportunity by getting into a pointless discussion about the use of percentages in datasets, so instead I placed more emphasis on an example of YouGov’s secretiveness that I consider to be much more significant.  Janta-Lipinski’s reaction was dismaying and fascinating in equal measure.

Like all online polling firms, YouGov weight their sample by recalled vote from 2011.  This is to ensure that the sample is politically representative, and doesn’t include too many Tory voters, or too few SNP voters, or whatever.  But YouGov take an additional and rather eccentric step – they split SNP voters, and SNP voters only, into two distinct groups and then weight them separately. 

This seems to be the most plausible explanation for why YouGov routinely produce much more No-friendly headline numbers than other online firms.  One of the two SNP groups is always upweighted dramatically, while the other is not. 

The likelihood is that YouGov are artificially giving far more weight to the type of SNP voter who is likely to vote No – but we don’t know that for sure, because the firm have persistently kept the voting intention figures for the two groups under wraps.  Instead, all we see in the datasets are the combined voting intentions of the two SNP groups after the weighting has been applied, which tells us nothing at all.

So I essentially laid down a challenge for Janta-Lipinski – if he truly believes that YouGov have no issues with a lack of transparency, would he undertake to publish the voting intention breakdown for the two SNP groups, and if not, why not?  Bearing in mind that he had just demanded an explanation from me, it didn’t seem unreasonable or unfair to raise a query with him. 

But suddenly he clammed up, and started working his way through a repertoire of flimsy excuses for why he couldn’t possibly answer the question right away.  After I pressed the point further, he eventually haughtily informed me that the only reason we wouldn’t be given the information is quite simply that we aren’t entitled to it – YouGov don’t provide “thousands of potential xtabs” at the “behest of bloggers”.  And this was from a man who couldn’t conceive of YouGov ever being accused of secrecy!

At that point I knew I was dealing with people who were withholding information for a reason they didn’t want to admit to.  Janta-Lipinski’s implication that this is merely one trivial detail out of “thousands” is plainly risible, especially in the light of Kellner’s blog, which specifically cites the splitting of SNP voters into two groups as a primary reason for thinking that YouGov are getting it right and others are getting it wrong. 

So what is the real explanation for the obsessive secrecy?  My suspicion is that YouGov know that if they showed the huge disparity between the voting intentions of the two groups, the strangeness and artificiality of what they’re doing would start to hit home, and questions would be raised about whether the small ‘No-friendly’ SNP group that is always so sharply upweighted is really typical of the huge section of the population that it is supposed to represent. 

The more a sample has to be upweighted, the greater the potential for error.  There’s also the question of why such a substantial upweighting is always required – couldn’t that be an indication that the basic assumptions being made are misconceived? 

None of this necessarily means that YouGov’s headline numbers are wrong, but it does mean they’re walking a tightrope and privately keeping their fingers very firmly crossed that it will be all right on the night.  Pretty much like all the other pollsters that Kellner looks down his nose at, in fact.

Janta-Lipinski did go on to indicate to me that he might eventually answer the questions he had evaded, but only after Kellner’s blogpost had appeared.  So I left a comment on Kellner’s post asking three specific questions –

1) Why the obsessive secrecy about the voting intentions of the two SNP groups?

2) If there is logic in splitting SNP voters into two groups and weighting them separately, why aren’t other potentially distinctive groups also separated out?  Doesn’t the current inconsistent approach smack of an organisation that is working backwards to generate the headline numbers that “feel right”?  (Something that Kellner is scarcely capable of being objective about, given his absurd declaration three years ago that the referendum was unwinnable for Yes.)

3) Given that we know a previous YouGov referendum poll was way out of line with the 2011 census in containing far too many English-born people and too few Scottish-born people, what steps have been taken to correct for this bias which is bound to skew results towards No?  If no corrective steps have been taken, why not?

I regret to inform you that none of these questions have been answered as of yet.  YouGov’s Great Wall of Secrecy, Bluster and Evasion remains unbreached.


Read more from James Kelly at http://scotgoespop.blogspot.co.uk/

Comments  

 
# pomatiaH1 2014-07-03 17:33
How long will it be before newspapers and the BBC start hyping up YouGov and casting doubts on the other pollsters!
 
 
# gopher3 2014-07-03 17:50
.Quoting pomatiaH1:
How long will it be before newspapers and the BBC start hyping up YouGov and casting doubts on the other pollsters!


Probably sometime this weekend.
 
 
# Nebulosity 2014-07-03 17:48
Would this be the same Peter Kellner who was one of the talking heads on BBC’s European election results programme claiming that the SNP were miles behind UKIP and had no chance in the race for Scotland’s final seat almost as soon as the polls closed? Think he and his bias just make up the results first…and then do the “blow to the SNP” number crunching afterwards.
 
 
# RobQos 2014-07-03 18:14
We all know what can happen in 2 months! Do polls genuinely reflect what people are thinking or are they used to try and influence peoples thinking?

en.wikipedia.org/…/…
 
 
# cynicalHighlander 2014-07-03 18:20
Quoting RobQos:
We all know what can happen in 2 months! Do polls genuinely reflect what people are thinking or are they used to try and influence peoples thinking?



The later I suspect Peter Hitchen wrote an article on it a year or so ago. The Union is getting desperate.
 
 
# Talorgan 2014-07-03 21:35
Quoting cynicalHighland  er:
The later I suspect


Pity

Otherwise we could just leave them in their false sense of security.
 
 
# Teri 2014-07-03 18:27
YouGov never give me polls on political matters any more. I feel that this might be because they know from previous polls that I am an SNP voter.

I can also say that when I did get political surveys to do I was usually thrown off well before the end of the survey; on one occasion I had only completed 26%. No explanation is every given for this and that makes me highly suspicious.
 
 
# gus1940 2014-07-04 06:14
Me too.
 
 
# Muz 2014-07-03 18:44
I enjoyed reading James’ interaction with the YouGov employee. It’s on James’ blog.

It comes across like Dot from EastEnders was running the shop while the boss was in The Vic.

It seemed to me that the YouGov employee was quite prepared to say anything to divert attention away from the issue James was trying to discuss.
 
 
# Diabloandco 2014-07-03 18:56
Yougov asked once how I had voted in the election of 2007 and thereafter I was only ever asked about technology – which , if you knew me was risible.
I stopped responding to their surveys after a couple of months . I am disinterested in their findings in much the same way as I am disinterested in the burblings of the MSM .
 
 
# neoloon 2014-07-03 19:01
Very interesting article.
It seems the fug of desperation which hangs over the No campaign is spreading over supposedly neutral organizations like YouGov.
I believe most people in Scotland are, at the moment at least, keeping their cards close to their chest – that approach being echoed in the opaque polling.
I’ve never been polled but I’ve made up my mind to lie if I’m ever asked in the future.
 
 
# Jacque De Molay 2014-07-03 19:11
To be honest I’m quite content to let Peter Lellner delude himself. The polling done by BT, RIC, SNP and other covers much larger numbers and the “feel” right against what is being said on stalls / doors and meetings.

Peter will throw his arms in the air and wail “feels right!”

It is normal practise to adjust your model when it is producing outcomes at odds with other evidence (polls).
However Peter has adopted that well known scientific policy – “you are all out of step with me”
 
 
# DisgustedTunbridgeWells 2014-07-03 20:02
Already been used by the London Evening Standard, no less than three times in Tuesday following their indyref ‘debate’.

standard.co.uk/…/… Only it wasn’t a debate at all as not one of the 500+ crowd changed their opinion – place was full of tartan Tories.

YouGov seem to supply them with boil-in-the-bag proNo stories e.g. standard.co.uk/…/…

But they rather like the sound of their own voice do YouGov, this one on how London is showing he rest of the UK the way on economic recovery is a belter: “The recovery has very much been like the Hunger Games, with the capital and surrounding areas burgeoning while the regions wither on the vine,” said Stephen Harmston of YouGov
 
 
# jdman 2014-07-03 20:22
I do yougov polls online a lot and only after it was pointed out that they drive people into a response that delivers their desired outcome did it become obvious to me they ARE fixing the outcome of their polls,
the questions asked prior to the BIG ONE are designed to corral the person into an answer which will give them the response they want.
I wouldn’t trust a yougov poll as far as I could throw it.
 
 
# DisgustedTunbridgeWells 2014-07-03 20:41
BTW have a pal at one of the main polling companies in London who says they did some work on voting intentions in the General Election. They found that IF there was a No vote then Labour would be annihilated in Scotland.

Labour would get the backlash and SNP would romp it, so it would definitely return the Tories no matter what. His advice to any Labour member was hitch yourself to Yes because if you’re seen a proNo then you may well find yourself out of a job in a couple of years time.
 
 
# Barbazenzero 2014-07-03 21:20
James,

Well said above, on your blog and on Kellner’s blog, but I’m surprised you don’t link to Survation’s new “Response To Yesterday’s Times, YouGov Articles and YouGov’s published research about Survation’s Scottish Independence Methodology” – survation.com/…/ – which also makes some excellent points on why YouGov may not prove omniscient.

By Kellner (aka Baroness Ashton’s consort) throwing down a gauntlet to other pollsters, we must support anyone but YouGov.
 
 
# Intransigent 2014-07-03 22:32
The most egregrious example of YouGov’s arbitrary weighting was, of course, before the 201 Scottish Election. Their polling found the SNP well ahead, but they decided their sample included too many SNP supporters, so they cut some out with some extra ‘weighting’, and got a slim Labour lead. Kellner admitted, close to polling day, that the SNP might be ahead, but that wasn’t certain. In the event, the SNP’s landslide victory reflected a lead even greater than the one YouGov originally found and discarded because they didn’t want to believe it.
 
 
# gregalach 2014-07-04 07:54
I joined yuogov polling earlier this year. For a while I lied about voting intentions, then gradually revealed a shift to yes. When I declared for yes and snp, I was dropped and now only get surveys about household goods, shopping habits etc, which barely apply to me on my tiny income. There is no doubt they are manipulating results. Their latest survey is to do with armed forces and I was dropped after first question. maybe changes in voting intention after AF day?
 
 
# gregalach 2014-07-04 08:17
ot by the way, tried to contact newsnet re the yes saltires which you are selling, but your contact systm didn’t work. I donated to the flags ansd expected to get one, but they took so long to be sent out that I have moved address and all attempts to contact the organiser, Mark Piggot have failed. I cant afford to buy a replacement. Very very disappointed and I am put off crowd-funding now.
 
 
# IXL 2014-07-04 14:26
See attachedlink to blog in FT
Yup – guilty as charged !

blogs.ft.com/…/…
 
 
# Alien Act 2014-07-04 15:21
Kellner may have Labour connections but you only have to google yougov to find out how many tories are involved in it, own shares in it, or help run it.
 
 
# davidredero 2014-07-05 00:14
I’m a bit surprised at some of the comments here. It’s perfectly possible YouGov are getting it completely wrong – and raising questions about the methodology here seems fair enough to me – but it’s going a bit into left field to think that they’re intentionally getting it wrong to try and influence the referendum.

I don’t even see why inflating the No vote would help Better Together in the first place. If anything it would hurt their campaign – when people think their side is going to win easily they have less incentive to come out and vote.

I really hope we’re not just bashing YouGov because they happen to be giving less optimistic results. Nobody really knows who is getting it right ultimately – and we won’t know that until the actual vote – so the best strategy is simply to look at trends rather than headline figures. If you look at the trend there’s reason for optimism even in YouGov’s figures, so I don’t see what good accusing them of bias does.
 
 
# GrayLiz 2014-07-05 12:40
The machinations of the YouGov poll ionsurprises me not at all. A long standing respondent to YouGov polls for a time asked me about my past voting pattern had been, did you vote in the 2010 General Election and how did you vote; how certain are you to vote in 2015 and how will you vote. My responses were that I had voted in 2010 and had voted SNP and I was also certain that I’d vote in 2015 and would vote SNP. And then suddenly I was no longer included in the political polls although I continue to be invited to participate in other polls. This would put me into the category of a committed SNP voter. And it made me wonder why I was no longer included in such polls. Seems like it was a deliberate ruse to skew the results of YouGov polls and if this exclusion happened to me as a committed SNP voter then it could have happened to many other committed SNP voters. The obvious result would be skewed in favour of a “No” vote in the upcoming Referendum.
 
 
# GrayLiz 2014-07-05 13:38
As an addendum to my previous comment I may say that I am regularly surveyed by Populus, Survation, Mori and New Vista on the my voting intentions in the same structure mentioned in my previous post.
 
 
# Jamieson 2014-07-05 14:46
I don’t believe ANY of the polls from polling organisations. They are not close to the results being found on doorstep canvassing, newspaper readers’ polls, and social media polling, where YES already show a good lead. Yes there are faults with those methods but there are flaws in the accredited polls too. But more importantly canvassing returns were far more accurate about the outcome than any of the Pollsters for the 1997, 2007, and 2011 votes.

Also, the volunteer populations (vp) from which the 1000 samples are drawn are people who by their actions in volunteering are clearly political anoraks.They are not representative in my opinion with the actual population who will vote in the Referendum. There are too many middle class, too few working class, and probably too many English people among them. And weighting thereafter can only do so much to redress the imbalances.
 

You must be logged-in in order to post a comment.

Banner

Donate to Newsnet Scotland

Banner

Latest Comments