Robyn Benson, PSAC

Lies, damned lies and statistics


Imagine you’re employed as a watchmaker. One day your employer collects all your tools and gives you a hammer and screwdriver instead. Brilliantly, you manage to fix maybe one or two watches. “See,” says the employer. “You didn’t need all that fancy stuff!”

That’s pretty much the position Statistics Canada was placed in by the Harper government when the fine-tuned long-form census was replaced by the voluntary National Household Survey. The former used sophisticated sampling techniques, permitting an accurate analysis of social and economic trends down to the community level. The latter used information from whoever happened to reply. The results? “The NHS slapped blurry goggles on our vision,” says one senior economist.

Let us pause for a moment to mourn the mathematical illiteracy of Christian Paradis, when he was still Minister of Industry this past May. From the horse’s mouth: “[M]ore Canadians responded to the National Household Survey than its predecessor, the mandatory long-form census.”

Well, uh, sure. The long-form census was sent to 20% of Canadian households. The NHS was sent to roughly 30% of Canadian households. The census had a response rate of 93.5%. Only 68.6% responded to the voluntary NHS. But yes, slightly more Canadians took part in an unscientific survey sent to 50% more households. Bravo, Minister.

And then a sideswipe: “Our government is committed to collecting statistical data while protecting Canadians’ privacy.” Did the previous long-form census fail to do that, as he suggests? Statistics Canada, in fact, has an unparalleled record in preserving the privacy of its sources—in fact, it’s mandated by law.

But on to the survey itself. What do people in the statistics business think of it?

“In some communities,” notes statistics professor David Bellhouese, “response rates were as low as 25 per cent, and a few even had a response rate of zero. Due to low response, data on First Nations was statistically compromised, and immigration statistics, which indicated that people from the Philippines were the largest segment of recent new arrivals, is flatly contradicted by Citizenship and Immigration’s own figures.

No wonder Statistics Canada, painfully aware of the shortcomings of the survey, attached a warning to its report that resembles, in longer form, the ones on cigarette packages—a “use at your own risk” message:

When comparing estimates from the 2006 Census long form and estimates from the 2011 National Household Survey (NHS) users should take into account the fact that the two sources represent different populations. The target population for the 2006 Census long form includes usual residents in collective dwellings and persons living abroad whereas the target population for the NHS excludes them. Moreover, the NHS estimates are derived from a voluntary survey and are therefore subject to potentially higher non‑response error than those derived from the 2006 Census long form.

When comparing income indicators from one source to another, users should be aware that the methodology of how the information was collected, the concepts used and response patterns can affect the comparability of income information. Given the sensitivity of most income indicators to such methodological differences, users should use caution when comparing income estimates from the NHS to other household income surveys, administrative data or 2006 or earlier censuses. In this analytical document, no comparisons to other data sources are presented. [Emphases added]

The NHS also indicates a significant rise in poverty in Canada, even despite the fact that the response rate from low-income wage-earners in such surveys tends to be relatively low. But the truth is that we really don’t know whether poverty is rising, decreasing or staying the same. Differing methodologies in the collection of data make the NHS unusable for comparison purposes, making it impossible to plot trends one way or the other.

The upshot of all this is that the data are too unreliable for the meaningful planning that the older, far more reliable census data permitted.

In 2011, then-Justice Minister Rob Nicholson defended a heavily-criticized tough-on-crime bill by saying: “We’re not governing on the basis of the latest statistics. We’re governing on the basis of what’s right….”

Now the government has confirmed that pesky matters like skewed data and low response rates won’t deter it from its ideological course. A near-universal chorus of disapproval from experts in the field, and the resignation of the Chief Statistician, didn’t matter a whit. Armed only with an invincible sense of its own righteousness, it directed Statistics Canada to collect flawed data and issue a flawed report. And it only cost the taxpayers an extra $22 million.

comments powered by Disqus

About this Entry

This page contains a single entry by Robyn Benson, PSAC published on September 17, 2013 8:29 AM.

Happiness is a strong labour movement was the previous entry in this blog.

Bangladesh: profit before decency is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Subscribe to feed Subscribe to this blog's feed / S'abonner au flux de ce blog

This blogsite is intended to be a safe space for everyone to comment on the issues raised here. Spirited debate is encouraged, in the language of your choice, but profanity, discriminatory words and other forms of abuse are not. Feel free to speak your mind, but please keep it civil and respectful.

Ce blogue a pour objet d’offrir un espace sécuritaire où chacun peut réagir aux questions soulevées. On encourage les débats animés, dans la langue de votre choix, mais non les insultes et les propos discriminatoires. N’hésitez pas à donner votre opinion, mais faites-le de manière civilisée et respectueuse.


Powered by Movable Type 6.0.7