Banner

  By a Newsnet reporter
 
On Sunday, details of a poll emerged in the media that suggested support for independence amongst 14 to 17 year olds was a mere 21 per cent.
 
In a further blow to the Yes campaign, those against independence was said to be running at over 60 per cent.

It was the main topic of discussion on Sunday's Politics Show and headlines followed.  Delighted Unionist politicians issued press releases as their pro-independence counterparts grabbed hold of an aspect of the survey that revealed over two thirds of respondents admitted they could well change their mind.

Unusually for such polls, the methodology and questions were published immediately and at first glance it seemed all was as reported – a blow to the Yes campaign indeed.

However, closer examination of this survey revealed, if not a disturbing aspect of the survey, then certainly something that seemed to cast doubt on the reliability of the poll.  The proportion of households that could, for want of a better description, be termed pro-Union, significantly outnumbered those households one would have said was pro-Independence.

The difference, which was over forty per cent in favour of pro-Union, was alarming. 

To clarify, the system used in this poll was RDD (Random Digit Dialling) which is pretty much as it sounds with random telephone numbers being dialled.  Before teenage respondents were asked their views, those of their parents or guardians were sought.

Of the 1018 households whose responses made up the survey, a whopping 594 had parents or guardians who said they would be voting 'No' in the 2014 referendum.  By contrast only 178 households had parents or guardians who said they would be voting 'Yes'.

So households that could best be described as anti-independence made up 58.3% of the survey, whilst households who could be described as pro-independence made up a mere 17.5%.  Undecideds constituted just over 24%.

We wondered about this apparent anomaly, so sent our questions to the contacts given on the published PDF.  We asked about the makeup of the survey and what we felt was an over-representation of anti-independence households which could have led to an inflated result for those opposed to independence.  We suggested that a more accurate survey would have ensured recent opinion polls by reputable organisations could have been used in order to eliminate such an obvious bias.

We are at present looking into the methodology used and would be grateful if you could confirm our initial belief that what can best be described as 'pro Union' households (if parent views are used) made up almost 60 per cent of respondents against around 20 per cent 'pro Independence' households.

This, to us, would potentially lead to an inflated support for No, given the survey found that 75% of No respondents took the same view as their parents.

If our figures are accurate, do you agree that the poll methodology ought to have mirrored more accurately, in terms of household view, recent polls which indicate the gap between Yes and No to be significantly less than the 35 to 40 per cent gap that your own sample appears to have assumed?

The reply was, to say the least, interesting:

The sample was a random sample of households with children aged 14-17, and the randomness ensures that the sample is likely to be broadly representative of the population of such households.  There have been no published surveys of the views of adults in such households, and it cannot validly be inferred that the views of such adults would be the same as the views of adults in the population as a whole.  To have biased the sample to do what you suggest - mirror recent polls of the whole adult population - would have been to have distorted the representativeness with respect to the young people who were our target population.

It seemed a strange response, that to have sought to ensure the ratio of households chosen did not exaggerate - one way or another - would have "distorted the representativeness of the survey".

We challenged the claim that the survey was representative:

Further to your latest response. We note your claim that the randomness of your methodology will produce a representative response.

Given that the parent replies very clearly did not produce anything like a representative response then we would ask you to justify this claim in relation to the results of the young person survey.

The response was even more interesting.

Randomness would produce an approximately representative response of the RELEVANT population, which here - as I said - is parents of people with 14-17 year old children, NOT all adults.  Since we don't know the views of such parents from any other source, there is no basis for your claim that representativeness has not been achieved.

So, the survey was seeking to represent only the relevant section of the population, which was of course households in which lived teenagers aged 14 to 17.  The views, according to the spokesperson, of these parents are not known.

Were the survey organisers claiming that households in Scotland with teenagers between the age of 14 and 17, even more likely to vote 'No' in the referendum?  If so then it was as big a story as the young person's survey.

We pursued the issue:

Then you appear to conclude that households with teenage children are even more likely to back a No vote than current polling suggests.

58.3% No compared to 17.5% Yes. 

Can you confirm this?

The response appeared to suggest a casual disregard for the views of the adults who had responded, the survey was not about adults.

All that can be said is that the adults whom the survey talked to had the distribution of views that you calculate from Table 10.  Remember that there is no such thing as a 'household' view, since the adults in the household will differ in their views, just as around 40% of the teenagers differ from that of the adult who was interviewed in the survey.  This was not a survey of parents, but a survey of teenagers.  We are not making any claims at all about the views of 'households', or of 'parents': as I said in an earlier email, that is a population about whom not much specifically is known. A different research design would be required to be able to assess the views of parents (or indeed other adults in a household), and how they do or do not differ from each other.

We sent one final email

Thanks for taking the time to respond. May we ask whether publication of this survey was always intended and whether the BBC or indeed any other news outlet has sought clarification on the methodology?

Also, does your organisation accept that the parent responses in this survey are exceptional when compared to adult polling results published by recognised organisations such as Yougov, Ipsos Mori and Panelbase.

If so, are you at all concerned by this difference?

The response is published below:

Publication was always intended: that is a condition of grant from the ESRC, and in any case is standard practice by academics. Indeed, we are grateful that you acknowledge that we have placed details of the survey in the public domain. The BBC did of course discuss the methods with us. The methods were also scrutinised in the usual rigorous way by the ESRC's peer-review process before they awarded us the grant. I have already answered your second main point: the results differ from the views of all adults, but that is not the relevant population for comparison in any assessment of how representative the views or the sample is of 14-17-year-olds in general.

As I explained, and as you kindly and accurately reported, the relevant population for comparison is not all adults at all. You claim also that we have not considered weighting the results. You did not ask me that, and the answer is that we have indeed examined in detail the effects of weighting. We will be reporting the results of the weighted analysis at a seminar in Edinburgh on 5 June, which is free for anyone to attend. You can sign up at:  http://www.aqmen.ac.uk/youngscotsurveyresults. This link was also made available on the BBC web site from the time of publication on Sunday 2 June. You would be very welcome to come. The main point we will be reporting there, however, is that no form of weighting makes much difference to the balance of views expressed by the young people, especially to the proportion intending at present to vote No.

The poll may or may not be accurate.  Young people in Scotland may well be even less likely to support independence than older members of the electorate.  We accept that households also may contain adults with differing views.  However what is known is that those adults who answered the initial question were overwhelmingly anti-independence, and that those youngsters who gave 'No' as their answers were in agreement with this adult in three quarters of the responses.

There are many factors which could have led to these young respondents giving the answers they did.  That they will have been influenced by their home environment seems the most obvious.  For a survey to refuse to acknowledge the importance of implementing a weighting system that acknowledges this home influence is incredible.

What other variables are at play in this survey is anyone's guess – how many people refused to take part when confronted by a random telephone call, how many young people gave answers as their parent listened in or felt compelled to placate their parent?

There is another more worrying aspect of this survey that no-one seems concerned about.  Teenagers are influenced by trends and a desire to be accepted as one of the crowd.

The publicity surrounding this poll will have the effect of telling these young people that anyone supporting 'Yes' is in the minority, and by a considerable margin.  The desire to conform and join the 'popular' group, for some, may well prove irresistible.

It is also not beyond the realms of possibility that what we adults accept as the normal cut and thrust of tribal politics, morphs into bullying in the less mature environment inhabited by fourteen year olds now that they know the 'winner'.

At the time of writing, the PDF document containing the questions can be read here.

This survey was funded by the Economic and Social Research Council which is based in Swindon.  According to its own website, it receives most of its funding through the Department for Business, Innovation and Skills, a department of the UK government.

"Apart from presenting the results themselves, the outcomes of the research will be used to develop teaching materials for schools across Scotland to have materials that are relevant for young people in informing the debate." the survey report concludes.

More work needs to be done, and quickly, in order to establish whether or not this poll is accurate.  Sadly, even if more surveys are carried out that challenge this poll, then the damage has probably already been done.

Comments  

 
# tartanfever 2013-06-03 21:33
I'm interested in the RDD (Random Digit Dialling) .

The survey claims that representation was sought across the different regions of Scotland, so some input must have been placed into dialling certain area codes.

Secondly, does that it actually mean that a computer makes up phone numbers ? In which case there must be many numbers that don't exist as actual phone lines.

Or is it that there is a database of phone numbers from which they choose randomly ? If this is the case, then what database are they using ? - local phone books, people that have previously signed up to surveys, just who exactly ?

With Better Together recent mobile phone text message stunts I wouldn't be surprised to hear that their database was accessed.

I would have asked for clarification on how RDD works exactly and how it's been used in this case.
 
 
# Jo Bloggs 2013-06-04 07:40
tartan

I'd imagine the database would be drawn from census results, as they were targeting families with children of a specific age.

I don't like the outcome of the survey, but from what I've read above I reckon their arguments are, on the face of it, valid.
 
 
# Garve 2013-06-03 21:41
Is RDD restricted to landlines? Do a significant number of households do without them these days, and if so, are there demographic patterns which would skew the results?
 
 
# rodmac 2013-06-03 21:52
Excellent Investigation Newsnet...and I have referenced it in my article here.

.../how-low-can-they-go
 
 
# springster 2013-06-03 22:13
Irresponsible to publish this poll, given the ages of the respondents, which leads me to believe there's more to this than meets the eye.

The ratio of anti-independence households to pro-independence households is glaring. The result itself was so out of step with recent opinion polls that alarm bells should have been sounding.

Impressionable teenagers will be affected by this survey and its very high profile media coverage. Younger voters have now joined 'women' in the difficult to reach category so the media machine is telling us now.

It's propaganda in everything but name, but that is the game we are in and we just have to get used to this. It's a Unionist media machine.
 
 
# millie 2013-06-03 22:19
Following on from rodmac’s comment-

Apparently, Better Together is sending ‘Teacher Resource Packs’ to every school in Scotland over the summer.

holyrood.com/.../...

Parents should be aware of this.
 
 
# Will C 2013-06-03 23:25
As a retired teacher (early), I find this tactic by the NO Scotland campaign insidious. I would hope and am pretty sure that the Scottish Government can put a stop to this potential manipulation of young minds. If not the EIS,SSTA and NASUWT should refuse to handle such black propaganda.
 
 
# Jamie Black 2013-06-04 12:13
millie - according to a poster on here, he/she has already been in high schools talking to children and persuading them to vote Yes. The poster used an example of a young adults first job - how you wouldn't expedct your parents to take your money and only gievn you some back.

Do you think that type of behaviour is acceptable? How did this person get intoa school to educate kids about the merits of Yes? Was it 'yes' sanctioned? A lot of questions, but I think the Yes campaign need to be wary, because it sounds liek they are doing the same, jsut in a different way.
 
 
# millie 2013-06-04 21:31
A pupil has the option whether to approach a ‘Better Together Stall’ or a ‘Yes Stall’.- and if they are both invited into a school for a balanced debate, that’s fine. (I’m sure the previous poster will have been invited by the school).

The final decision regarding the distribution and contents of referendum material for use by teachers in the classroom (where pupils are a captive audience) should be made by ‘Education Scotland’.

My own preference would be that the pupils in question are given a short ‘general’ political awareness course (approved by Education Scotland), followed by several question and answer sessions with representatives from ‘Yes’ and ‘Better Together’.

The pupils should be encouraged to grill each side, and then come to their own decision.
 
 
# fend 2013-06-06 00:13
Millie - I have addressed JB's issues in the original thread.
newsnetscotland.com/.../...
Yes and No we're invited by the school to setup adjacent stalls as part of a topic week. We were there answering questions for a couple of hours. The "be independent and take control of your own purse strings" example gave the pupils a good insight as to the key issue. The No stall were left telling pupils Scotland would have no army...
 
 
# Leader of the Pack 2013-06-03 23:22
I don't know why anybody thinks this is any different to any other unionist commissioned poll. There are at least a dozen easy ways to obtain a desired result from a seemingly random polling strategy and the No campaign have employed the lot. Its no coincidence that the last Scot election saw the polls results shift cataclysmically inside a single week from 1 extreme to the other. These polling tactics are well documented in the US elections and are favoured by the Republicans and Neo Cons. I honestly believe the true level of support for Yes and No is pretty evenly balanced just now with a large group of undecided or indifferent. The polling is just another version of their scaremongering and negativity.

All the major polling companies have direct links with Westminster employing ex political staff of one party or another. There isn't a single mainstream polling organisation in this country which isn't politically tainted.
 
 
# GogsyBroon 2013-06-04 06:42
If this is just a unionist sponsored poll (as you claim), then why aren't the Nats commissioning their own polls to refute this? Would it be because the poll has reflected an uncomfortable truth that the wider electorate don't believe in the vision of independence peddled by the SNP?
 
 
# Alba4Eva 2013-06-04 08:35
Quoting GogsyBroon:
If this is just a unionist sponsored poll, then why aren't the Nats commissioning their own polls to refute this?


Because no media (BBC or newspapers) would publish the results... If they did, it would be in the proverbial small print of said media.
 
 
# Leader of the Pack 2013-06-04 14:59
Quoting GogsyBroon:
If this is just a unionist sponsored poll (as you claim), then why aren't the Nats commissioning their own polls to refute this? Would it be because the poll has reflected an uncomfortable truth that the wider electorate don't believe in the vision of independence peddled by the SNP?


No much point in showing up the negativity and dishonesty of the No campaign if we're going to employ the same methods ourselves now is there? There is only one version of Independence and the whole world employs and understands it including the UK Government who understand it to the point of forcefully employing it even when its not wanted or asked for.
 
 
# Mac 2013-06-04 06:21
It is known that RDD leads to some systemic biases. Pollsters using RDD polling, ICM and Populus, both find too many Labour voters in their raw samples and have to use weighting to correct it.

This Edinburgh University poll is clearly unrepresentativ e and it is odd that the researchers are claiming that different weighting methodologies make no difference to the results. It should.

We know that all polls show that the young, of all the age groups, are more likely to support independence. This is the first poll that marginally shows the opposite.

That in itself is a very strange result.

Sometimes researchers torture the data to get the results they want. This is the first time I can recollect where satisfaction is being drawn from systemic bias.

This is science but it is not science as we know it. The researchers may wish to reflect on that.
 
 
# cardrossian 2013-06-04 06:52
Stop worrying. Teenagers being teenagers will change their minds many times between now and polling day.

What matters is that the utterings of the Better Together mob are continuously and very publicly slammed, particularly on the social websites. Just like their school lessons, the teenagers will absorb the information and will vote sensibly when the time comes.

As for the poll. My daughter's house was not polled, where the views of two teenagers would have had to be counted. How would two teenagers of voting age in a house distort the findings?
 
 
# From The Suburbs 2013-06-04 07:36
Call Kaye discussing politics in schools.

Herald reports George Foulkes has referred Iain MacWhirter to Ofcom on basis he is only presenter of the STV programmes on Road to Referendum tonight.
 
 
# setondene 2013-06-04 07:53
I think teenagers go through a development process that leads them from seeking reassurance and stability (as in a stable home) to being rebellious and then seeking personal independence. A kind of parallel of political unionism (safety/stability) and independence. Their own personal views are maturing fast during the teenage years. Thus a fourteen year old could have different views entirely by the time she/he is seventeen. But people are right about the damaging effect of this poll - teenagers are group conformists of the first order.
 
 
# Old Smokey 2013-06-04 08:06
As has already been mentioned, George Foulkes has complained of Political Bias of a programme that airs tonight on STV at 8 pm.
This is typical of Foulkes, as its not about Bias, but about output not being part of the Labour/Unionist picture and being outwith the Labour/unionist control (the internet being an example of Labour/unionist angst, which coined the derogatory term 'cybernat'). Foulkes is unhappy that Ofcom refused to take up his complain - awe shame!
heraldscotland.com/.../...

news.stv.tv/.../...
 
 
# pmcrek 2013-06-04 10:01
Turns out not surprisingly that the responses to the survey amongst the highest economic grouping skewed the poll by around 200%. Only 13% of responses were from Scotlands lowest economic demographic.
 
 
# loveme2times 2013-06-04 10:14
Good article in the scottish review on the subject of scotlands young folk.

www.scottishreview.net/.../

While i was there also picked up on this piece by dr mat qvortroup

www.scottishreview.net/.../
 
 
# neoloon 2013-06-04 11:30
If unionists wish to indulge in such eager self-delusion then so much the better.
 
 
# Jo Bloggs 2013-06-04 13:02
Not if they're preparing to fix the referendum and using this to prepare the perception abroad in advance, so any protests can be more easily dismissed. Another reason why there should be international monitors involved.
 
 
# Mac 2013-06-04 11:46
Here is a stark difference;

Voting intention by highest parental educational attainment:

Edinburgh Uni (EU) survey v Scottish Government (SG) statistics for Highest level of qualification held by adults aged 16-64, Scotland 2010.

Higher education degree: EU 36%, SG 21%

Higher education without degree: EU 27%, SG 14%

Upper secondary: EU 24%, SG 26%

Lower Secondary: EU 13%, SG 39%

This is a massively skewed poll.
 
 
# gfaetheblock 2013-06-04 15:49
Is the inference here that those with a higher level of education are more likely to reject Independence? Do we know why that is?
 
 
# Abulhaq 2013-06-04 12:00
Random digital dialing is a relatively quick but flawed device for polling, even among adults. Prior notification, by mailshot, is usually necessary before initiating the poll. Given that the age group concerned rarely, if ever, uses landline phones for communication the result achieved is based on unscientific, if a poll can ever be scientific, methodolgy. Having your parents around doesn't help either.
 
 
# bunter 2013-06-04 12:17
Lunchtime BBC in Scotland News going big on the £40m spent in private hospitals due to a Bradford FOI, timed nicely to coincide with the terrible figures today from NHS down south. Will they mention that the figure is 0.5% of the budget. Will they mention that NHS Sco is outperforming its neighbour down south by 9% regards waiting times. Will they mention the £200m per year PFI bill from Labour? Watch this space......
 
 
# michaelkav 2013-06-04 12:42
Put out a bad poll and hope all the people voting YES will change to NO as the poll shows NO is way ahead. ^^ This Machiavellian crap does not work in Scotland these days. On the ground speaking to fellow Scots these numbers seem way off mark so I can only conclude it is Machiavellian in nature.
 
 
# Old Smokey 2013-06-04 13:12
Here is a must watch :
Its an interview by Russia Today's Max Keiser of George Galloway.
Frankly I found Galloway full of what comes out the backend of a cow!. He came out with corkers, such as , he was more concerned about the people of England having a permanent Tory Government (which of course is not true). He claims that the population of Scotland is falling (obviously missed the last census data); claims the coastline of Scotland is only 4000 miles and not the actual 6158 miles for the mainland ; claims that the Oil and Gas is fast running out ; Claims that the breakup of India, the USSR and Yugoslavia were big mistakes and is against the break up of his country (UK); claims socialism doesn't work ; claims that Scotland has very few resources.
Little wonder Galloway is seen as a clown
www.youtube.com/.../
 
 
# gerrydotp 2013-06-04 15:12
No offence but Is that the same Galloway who was in a black jump suit lapping milk on Big Brother that time?
Aye well, I don't think I'll bother if you don't mind.
 
 
# proudscot 2013-06-04 15:15
With such lack of accuracy in his myriad claims, it's a wonder Galloway hasn't been signed up as Johann Lamont's scriptwriter for her FMQs performances!
 
 
# Viking Girl 2013-06-04 15:58
Will C
I have serious reservations about the use of so-called 'teaching materials' from any source regarding, not just the forthcoming referendum, but any political matters whatsoever.
I hear that a local council has advised their teaching staff to keep their political views to themselves. I'm surprised that they felt they had to do that as it is self evident that any teacher would be unprofessional in discussing their own political views in the class-room.
I was surprised at the result of the poll. I would have thought that support for independence was high among the young.
 
 
# JS_scotland 2013-06-05 21:58
i40.tinypic.com/11mf9dy.jpg

Survey done at James Watt College, Greenock (Finart St Campus) 20 May. Straight yes/no. No option for undecided.
 
 
# InfrequentAllele 2013-06-05 22:32
49% Yes vs. 51% No

Much better for the Yes campaign than the previous student poll - I wonder if it will get the same coverage in the media? Probably not ...
 
 
# Radge 2013-06-26 11:38
Thanks for this. I was wondering how on earth it cost £180,000 to ask 1,000 teenagers how they intend to vote on one question.
 

You must be logged-in in order to post a comment.

Banner

Donate to Newsnet Scotland

Banner
Banner

Latest Comments