Don't Trust The Polls - The Systemic Issues That Make Voter Surveys Unreliable - US News - The Guardian

11/10/2016
Don'ttrustthepolls:thesystemicissuesthatmakevotersurveysunreliable|USnews|TheGuardian
Don't trust the polls: the systemic issues that

make voter surveys unreliable
You can expect a deluge of numbers as the presidential election approaches. Be cautious, though, as other
recent vote predictions have been way off the mark
Mona Chalabi Guardian US data editor
Wednesday 27 January 2016 15.26GMT
America loves polls. In the rst 26 days of this year, 186 political polls were released. Most of
them attracted some attention; some prompted hundreds of headlines. And its only going to
get worse.
By the time the country prepares to elect its next president in November, the news will be
awash with numbers. As a result, some Americans will feel condent they know what the
results will be before they have even cast a vote. The question is, should we trust what these
polls are telling us? My answer is a reluctant no.
And its not just because I was in the UK on election night last year when every single polling
company had substantial amounts of egg on their faces. There are ve reasons why we should
be more distrustful of political polls than ever before.
1. The media are fallible

You may have noticed this: a new poll comes out often a poll that was commissioned and
published by a media company and it gets heavily publicised by the media. In doing so, the
media often increase the name recognition of whichever candidate is in the lead. As a result,
that candidate becomes even more well known and gets an extra boost the next time people
are surveyed. Donald Trump was polling at 2% before he announced his candidacy.
Immediately afterward (while the media was dedicating 20% to 30% of candidacy headlines to
one candidate) his support jumped to 11%.
This is known as a feedback loop, and its even more important in the early stage of the
election cycle when the public is trying to spot someone they know from the throng of white
men in similar suits with similar haircuts (did you know there are more than 200 people vying
for the Democratic nomination? Can you name any of them whose surnames arent Clinton or
Sanders?).
2. Im fallible
Yep journalists, even statistically literate journalists, can be dumb. Its the reason why just
about everyone failed to predict Trumps astronomical rise. Thats not just because the
methodology of polling is itself awed (more on that below), its also because the analysis is
aected by the humans conducting it.
Personal experience and personal beliefs get in the way. Many of the journalists who dismissed
Trump with projections that he had a mere 2% chance of winning simply didnt know any
potential Trump supporters in their personal lives they didnt get it. And their personal
beliefs also encouraged a bit of wishful thinking they didnt want to get it.
https://www.theguardian.com/usnews/datablog/2016/jan/27/donttrustthepollsthesystemicissuesthatmakevotersurveysunreliable
1/3
11/10/2016
Until we can nd perfectly objective robots to conduct these polls, asking 100% neutral
questions and communicating them to you the reader with 100% neutrality, well, weve got a
polling problem on our hands. Humans, awed as they are, produce polls that are imperfectly
designed, imperfectly conducted and because of people like me imperfectly analyzed.
3. Predicting the future is hard

Polls are not a crystal ball but (at best) a decent snapshot of now. Even the wording of polls
reects this focus on the present rather than the future: If the presidential election were
today, for whom would you vote?
Consider how dierent your present and future answers would be to questions like: If you
had to eat your lunch now, what would you eat? or If you had to choose a romantic partner
right now, who would you pick? You might argue that food, romance and politics arent all
that similar, but the answers all point to a basic and consistent truth about human preferences:
things change.
This is why polls get more accurate as the future draws closer. And even then, there is no
opinion poll quite so reliable as the results of what people have actually decided to say in the
privacy of a voting booth on election day.
4. Its hard to contact people

In 2013, 41% of US households had a cellphone but no landline and that number is on the rise.
This poses a problem for polling companies because the 1991 Telephone Consumer Protection
Act means that they cant just autodial those cellphones. Not being able to autodial means that
it is incredibly expensive and time-consuming for companies to poll those people. Even more
problematic, this varies across America younger households and poorer ones are much less
likely to have a landline as seen in the chart below from Pew Research Center. So pollsters
need to take that into account when they are trying to take the pulse of the nation.
5. Most people dont want to be contacted

Response rates (thats the percentage of people who answer a survey when asked) have
plummeted. In the 1930s, it was over 90%; in 2012 it was 9%, and it has continued to decline
since then. Whats more, the US population has grown 2.5 times larger since the 30s so the
overall participation rate (the percentage of the total population that ends up actually
completing a survey) has fallen even faster.
In the end, youre left with about a thousand adults (if youre lucky) who are taken to be
representative of the approximately 225 million eligible voters in the United States. All of this
is well known to pollsters, who claim that their complex mathematical methods can correct for
these shortcomings. But rarely have I seen the question asked: What if a certain type of
person answers polls?
Whether they are cold calling or they have created a panel of individuals that they can
repeatedly survey, polling companies all face the same basic issue of how to incentivize people
to answer questions.
Some pollsters such as Pew Research Center pay respondents nothing, while others like
YouGov oer credits that can be slowly saved up towards gift vouchers. What if the sort of
people who are willing to spend hours with little or no reward have something in common?
What if they tend to be consistently more conservative? More liberal? Wouldnt that skew all
the poll results in a certain direction? Ive seen no research on this point, but it certainly seems
a potential aw worth considering.
2/3
11/10/2016
All of these things together mean that getting a random survey sample that is nationally
representative is incredibly dicult. Gone are the days of icking through a phone book and
calling the rst number you land on.
And these polling problems arent going away, in fact they are only getting worse with time.
Since the last US presidential election, polls have failed to predict outcomes of the Israeli
national election, the Scottish referendum and the UK general election. In every case, their
margin of error wasnt a few percentage points it was way o the mark. Most of what Im
saying here isnt any kind of groundbreaking revelation. So why, then, do we continue to rely
on polls?
In a world with so much uncertainty theres an emotional comfort in the coldness of numbers.
Were reassured when were told what will happen. Pamela Meyer, CEO of a deception
detection company, often gives the same advice about how to spot a lie: If you dont want to
be deceived, you have to know: what is it that youre hungry for? Journalists need to be more
conscious of what they want the polls to tell them before they start reporting them. And
readers should be cautious of any headline that condently tells them what will happen on 8
November 2016.
Since you're here ...

...we have a small favour to ask. More people are reading the Guardian than ever but far fewer
are paying for it. And advertising revenues across the media are falling fast. So you can see
why we need to ask for your help. The Guardian's independent, investigative journalism takes
a lot of time, money and hard work to produce. But we do it because we believe our
perspective matters because it might well be your perspective, too.
If everyone who reads our reporting, who likes it, helps to pay for it our future would be more
secure. You can give money to the Guardian in less than a minute.
Make a contribution
Alternatively, you can join the Guardian and get even closer to our journalism by becoming a
Supporter.
More blogposts
Topics
US elections 2016 US politics
Save for later Article saved
Reuse this content
3/3

Don't Trust The Polls - The Systemic Issues That Make Voter Surveys Unreliable - US News - The Guardian

Uploaded by

Copyright:

Available Formats

Don't Trust The Polls - The Systemic Issues That Make Voter Surveys Unreliable - US News - The Guardian

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Don't Trust The Polls - The Systemic Issues That Make Voter Surveys Unreliable - US News - The Guardian

Uploaded by

Copyright:

Available Formats

11/10/2016

Don't trust the polls: the systemic issues that

1. The media are fallible

3. Predicting the future is hard

4. Its hard to contact people

5. Most people dont want to be contacted

Since you're here ...

You might also like