Opinion Polls Are Only as Reliable as Their Design

Commentary

“Garbage in, garbage out” is a well-known adage that means the quality of the input determines the quality of the output in any system. For example, if a mathematical equation is improperly expressed, the answer is likely to be incorrect. The same applies to public polling. If the survey questions are vague or biased, if they are not given to a random sample of the larger population, or if they exceed the knowledge or experience level of those polled, the results are likely to be inaccurate.

These features even apply to seemingly straightforward behavioural surveys, such as election polls that ask people to choose which candidate or party they plan to support on voting day. Automated phone polls, the cheapest and most popular way such surveys are conducted, rarely ask respondents whether they plan to vote or if they are eligible to vote based on their age and citizenship.

Even more errors afflict opinion polls focusing on knowledge, beliefs, and values, which is why experts and ordinary citizens increasingly question their results.

There could hardly be a more contentious political and emotional topic in Canada these days than the status, rights, and well-being of the country’s indigenous people. This is why polling results about indigenous issues must be viewed with extreme caution.

This is my take on two recent Angus Reid Institute polls about indigenous issues, one conducted online between June 20 and 23 given to 1,619 adult Canadians, the other between July 24 and July 29 administered to 2,508 adult Canadians.

Yes, the survey results, as stated, may well have been weighted to represent adults nationwide. However, this does not guarantee randomness if the larger pool from which the sample was drawn was unrepresentative of Canada’s adult population. Unrepresentativeness seems likely because the samples were drawn from the Angus Reid Forum, a self-selected convenience sample of Canadians that anyone with an email address can try to join. In short, the “random sample” of those participating in the survey taken from a non-random cohort of Canadians, namely the Angus Reid Forum, is a misuse of representative selection.

It is not unreasonable as well to assume that the Angus Reid Forum is overrepresented by individuals with a higher interest in and understanding of public policy than ordinary Canadians, potentially introducing a level of bias in its use in survey research.

Nevertheless, the Forum’s collective views display a significant lack of knowledge of important indigenous issues. For example, when asked about the shocking May 27, 2021, Tk’emlúps te Secwepemc’s announcement heard around the world of the discovery of the buried remains of 215 children alleged to have been students at the Kamloops Indian Residential School, 55 percent claimed they knew nothing about this announcement or only “saw some media coverage and had the odd conversation about it.”

Unlike other variables like geographical region, ethnicity (indigenous vs. non-indigenous), gender, age, and political affiliation, differences in participant knowledge were never weighed against related poll questions, including this one:

“The Kamloops band’s claim of 215 unmarked graves of children was later revised to about 200 ‘anomalies’ and suspected burial sites. The federal government allocated $12.1 million in funding to support the investigation of this issue. To date, no additional reports have been made public. What is closer to your view?”

Sixty-three percent of respondents chose: “People should only accept the claim that this is evidence of unmarked children’s graves if further information is publicly available to verify it through excavation.”

However, it is unknown how the views of the 55 percent who knew little or nothing about the Kamloops issue affected these results by cross-tabulating them against these less informed individuals.

Reinforcing its inherent knowledge-based limitations, the survey also found that most Canadians overestimated how many indigenous children attended residential schools. Roughly a third of respondents thought it was 40 percent to 60 percent of children, while another third chose 60 percent to 80 percent. Some thought it was even higher.

This is more evidence of “garbage in, garbage out” because only about 30 percent of indigenous children spent less than five years attending residential schools.

Data released on Aug. 20 from the second poll suggested that 55 percent of Canadians feel that indigenous people have a unique status in Canada, while 45 percent believe “they have no special status that other Canadians don’t have.” The 45 percent of deniers is astounding given that the “special constitutional relationship” between indigenous peoples and the government of Canada is codified in Section 35 of the Constitution Act, 1982, which recognizes and affirms indigenous rights that predate the constitution by over 200 years.

The establishment of Section 35 of the Constitution was not the end of the legal evolution of indigenous rights, which have been further augmented by the Supreme Court in cases such as R. v. Calder, R. v. Sparrow and R. v. Gladue.

If so, many presumably well-informed Angus Reid Forum members are unaware of these special rights, how much less must other Canadians know about this key indigenous issue?

Despite demonstrating gross disparities between perception (among survey respondents) and reality (according to history and law), polls based on non-random convenience sampling whose participants exceed the knowledge level of ordinary Canadians prove that the aphorism “garbage in, garbage out” is alive, if not well, in Canadian society.

Views expressed in this article are the opinions of the author and do not necessarily reflect the views of The Epoch Times.