A report by three high-profile academics with expertise in public opinion research who were commissioned to look into polling failures that marred the 2017 Calgary municipal election campaign sharply criticizes the role of the city’s media in covering and sponsoring the polls.
Christopher Adams of the University of Manitoba, Paul Adams of Carleton University, and David Zussman of the University of Victoria singled out Postmedia, whose Calgary Herald and Sun newspapers sponsored a controversial series of polls that wrongly indicated a conservative candidate was dramatically in the lead, for particular criticism.
Professors Adams, Adams and Zussman were also critical of the polls that contributed to the controversy, in particular three demon-dialler surveys done by Mainstreet Research of Toronto for Postmedia that put Conservative candidate Bill Smith far ahead of incumbent Mayor Naheed Nenshi in the October 2017 campaign, in one instance by 17 per cent.
“The panel heard that these polls, which received the greatest media attention during the campaign because of their number, their startling results and their association with the two Calgary dailies, significantly affected he course of the campaign,” the report says. “They threw Nenshi’s campaign on the defensive, gave impetus to Smith’s campaign, and possibly doomed the prospects of another candidate, Andre Chabot, who Mainstreet’s poll suggested was not a close contender.”
Nevertheless, on election day on Oct. 17, as is well known, Mayor Nenshi won by a margin of 7.65 per cent. In the wake of his election victory, some of the mayor’s supporters accused Postmedia of being part of an unsuccessful campaign to roust Mr. Nenshi from office through biased reporting that assigned undue credibility to flawed polls.
As the report put it, “the Mainstreet polls … triggered an acrid debate in the media and on social media, in which the Nenshi campaign attacked the firm’s motives and independent academics questioned its results and methodology.”
Meanwhile, all through the campaign, “Mainstreet executives responded with unshakable confidence in their results and attacked their critics, often in personal terms, at one point suggesting there would be ‘payback’ after the election results were known,” the report says.
The trio of professors released their 70-page report independently early this morning. It had been commissioned by the Market Research and Intelligence Association, a national polling-standards group. It was never published by MRIA, however, which unexpectedly announced last week it was in financial trouble and was almost immediately going out of business. MRIA ceased operation on Tuesday, July 31.
On Postmedia’s coverage, the independent panel’s members said, a review of media coverage showed “Postmedia in particular was not critical enough in its reporting of polls for which it was partly responsible.
“Moreover, Postmedia did not share with its own readers concerns it had about the polls and the degree to which Mainstreet was altering its methodology to address them,” they went on.
They found that while Postmedia and other media operations reported on the dramatic discrepancies between the Mainstreet polls and others published during the campaign, “that coverage was not technically sophisticated and would not have left readers fully equipped to evaluate the polls.”
The panel noted that two other polls, one done for a group that supported a light rail transit line and the other for a long-term Canadian academic project, were off base too, but not as dramatically as Mainstreet’s, and in the right direction in that they both showed Mayor Nenshi in the lead.
“The Mainstreet polls, powered by the dominance of the two city dailies, and the ricochet effect it had through its reporting in other media outlets and social media, created a dominant narrative for the campaign: that the election, in short order, had become a two-horse race between Nenshi and Smith, with the mayor struggling to stay competitive.”
The panel also said it “found that Mainstreet’s overconfidence and its contentious style of public debate significantly contributed to the embarrassment of the industry when the results were proven to be radically mistaken.” The authors added that the Toronto polling firm’s public confidence in its results during the campaign “contrasted with internal concern about its results that led to adjustments in methodology.”
Those technical changes, the authors said, made some problems worse – in particular Mainstreet’s difficulties contacting a representative sample of younger voters. “When the election proved to have an unexpectedly high turnout, especially among young people, these vulnerabilities were yet further exposed.”
The three professors made 10 recommendations, many of them tied to actions that should be taken by pollsters, Mainstreet in particular, in conjunction with MRIA – which is going to be a problem now, seeing as MRIA has ceased operations.
Recommendations that could still be implemented, even without a Canadian polling standards organization, included:
- For media to fully disclose their commercial and financial relationships will pollsters when obtaining polling data on the basis of exclusivity
- Application of “normal journalistic context and scepticism” when reporting on polls obtained through exclusive relationships
- Use of a standardized disclosure checklist that all polling companies would publish with their polls, including an accountability tool for use by the public
With particular regard to Mainstreet, the panel had recommended the company work with MRIA to choose “a neutral academic auditor” who could evaluate their practices and adherence to standards with the goal of welcoming them back into respectable polling circles. That will be harder now without any Canadian polling standards body.
A recent poll by Mainstreet of Alberta voters’ intentions said the United Conservative Party led by Jason Kenney is dramatically ahead of Premier Rachel Notley’s governing New Democratic Party, leading to much crowing on social media by UCP supporters.
Click here to read the full report by Dr. Adams, Mr. Adams, and Dr. Zussman.
“Online techniques face other challenges such as creating a representative list from which to draw respondents.” – MRIA report (May 2018)
These professed “online polls” that offers people prizes/money to participate should also be subject to close scrutiny. Unscientific online polls start with a badly selected sample. It’s not a sample of anybody — it is people who have volunteered/signed up to take online “internet” polls for incentives like prizes or money. They rely on an established directory of respondents for their online polls. They are not randomly selected because there is no complete list of voters’ IP internet addresses — therefore the poll is not probability-based and has no margin of error. Furthermore, there is no way such online polls can take into account the views of non-internet users.
As the 2019 election approaches and online polls start to become ubiquitous, peruse them with a high degree of caution. Next time you see a published online internet poll with this caveat (usually at the end of the poll) be very suspect of the results: “The survey utilizes a representative, but non-random sample, therefore margin of error is not applicable.”
Polls are a blunt-force propaganda tool, not a scientific instrument. We should treat them as such. The pernicious use of polls is eroding our democracy.
Poll results depend on the wording. Phrase the question differently, provide missing context, and you’ll get different results.
The regular failure of election polls should be obvious to everybody by now. Government by poll is even more disastrous.
As the classic saying goes, “mistakes were made”. The polling industry in particular, does not seem very interested in learning from those mistakes, or taking responsibility so I would not be surprised if the same mistakes are repeated again.
I think the main stream media is just as susceptible to repeat their mistakes too. They seem to have a conservative bent that was at odds with Mr. Nenshi and some animosity or antagonism developed between him and them over time, perhaps increased by personalities. Therefore it is not too surprising they started looking for a white knight to save their interests and the city, even if they had to partly create a bandwagon for him.
I don’t think the mainstream media has become any less vocally conservative since Nenshi was re-elected, if anything they have become more so. Now, with the disappearance of the MRIA there is even less constraint on them than before. Who will investigate and censure anyone for badly done polls now? The only saving grace is the loss of credibility for the mainstream media – next time the public is not going to be as easily fooled, at least not in Calgary. However, the fake bandwagon might still work like a charm in Red Deer or Fort McMurray to get some people to vote the way the pollsters and their media partners want.
When I was growing up in BC and becoming politically aware the government of the day (WAC Bennett’s Social Credit) banned all political polls during the writ period because of their concern they could be used to affect election results. Of course this led to alternatives such as the ‘hamburger’ and other similar polls.
In any case this report demonstrates how fragile democracy can be whether it is through polling companies pushing an agenda (mainstreet is not the only suspect polling organization) or foreign entities using social media to push an agenda.
Once upon a time polling could be done with a reasonable degree of accuracy. Almost everyone had a landline, so people could be called at random, and being asked to participate in a survey was a rare enough event that people were largely willing to do so.
I don’t think it is possible any longer. Landline based polls are hard pressed to get a decent sample of cell phone carrying young people, and people with landlines are tired of unsolicited calls, so they don’t participate.
Paying people to participate seems like a reasonable idea, especially since research companies are asking participants to give them something for free (their opinions) which the company will then refine into a product they subsequently sell. Unfortunately doing so would restrict their sample to people with low enough income the ten or twenty dollars offered to a prospective participant would be enough to get them to give up 15 minutes of their time. You really have to wonder what motivates people to participate in polls anymore, and the answer could be scary.
JE’s assessment of online polls is, in my opinion, spot on.
Polls may make interesting news stories, but they no longer have the accuracy they once had. Furthermore, like Geoffrey says, using them to form policy is dangerous.
So basically David the take home message I got from your article was that Mainstreet’s polling during the Calgary mayoralty race was way off so therefore the recent poll showing UCP support at over 50% must be wrong as well.
Personally in the end I think Nenshi benefitted to the extent that it inspired his supporters to work harder and get out and vote. As for the Alberta NDP, recent news that the TransMountain pipeline will cost an additional $1.9 billion and take a year longer to complete won’t help with popularity. Certainly a large part of the sales job by the NDP for the climate leadership plan focused on getting support for pipeline construction(didn’t happen) and now you have Jagmeet Singh planning to run in a Burnaby by-election on an anti TransMountain platform. So even Premier Notley’s own federal party is running against her. Interesting times to say the least. Enjoy your day
And then there’s the possibility that more and more Albertans are more progressive than the right wing thinks. Nenshi has been a reasonably good mayor, and is popular, particularly with the younger crowd who got him elected. We worked hard for our progressive councillor, inspired to action by a conservative attempt to sue her for a vote she participated in that helped shut down that conservatives development plans. It might come as a surprise to some, but bullying, bad mouthing, and one note whining about taxes being too high is starting to wear thin with voters.
As to that Kindermorgan pipeline….no one can turn a sow’s ear into a silk purse (remember that old metaphor?) not even the excellent government of Rachel Notley, not even the well intentioned government of Justin Trudeau. Economics are economics….and past a certain point…can’t be spun.
P.S. Rusty pipe is rusty pipe also…and after 65 years is likely past its due date. Too bad CAPP and other oil apologists succeeded in convincing Justin public money had to rescue it….but then, our tax dollars have been subsidizing Big Oil for decades…only difference now, is its public knowledge.
Can’t wait to see how the right wing pollsters spin that.
Clearly the polling industry has some issues. But polls can still be useful, if taken with a certain degree of critical thinking. Polling aggregators, like Nate Silver in the US or Eric Grenier at the CBC, probably give better information than single polls or single pollsters. But remember, even the best polls can only predict popular vote percentages, not riding by riding results. For a mayoral election, that might have high predictive value; for a Provincial or federal general election, how the vote breaks down by riding is harder to pin down in most polling, unless you conduct statistically valid random sampling in each and every riding, a logistical and financial task that is not for the faint of heart.
I do agree, though, that those online panels have no statistical validity at all. The selection bias alone is enough to flunk anyone out of grad school.
And still not a peep from the Calgary Herald. It’s as if this embarrassment never happened. They’d rather utilize their space demonizing the pension plans afforded some City of Calgary employees (having been tipped off by “the good folk at the Canadian Taxpayers Federation”).
The good folks at the Canadian Taxpayers Federation indeed…..all 5 of them! Could be time for someone to do an expose of astroturf organizations, and how the MSM uses them to create false ‘appeals to authority’.