Beware the Perils of Polls

A question's wording has a significant impact on the validity of poll results

 By Barbara McAdorey

According to a recent COMPAS/National Post poll on Canadians' attitudes regarding various freedoms, 78% of Canadians believe "women should have a completely free choice" in the matter of abortion. This figure contradicts most other polls conducted over the past several years which indicate only about a third of Canadians believe in a woman's right to unrestricted abortion.

Why the discrepancy? Because the COMPAS/National Post poll's question regarding abortion was ambiguous, leading, and biased, accounting for an inflated number of Canadians supporting a woman's right to abortion-thus the results are invalid.

With regards to ambiguity, the interviewer asked the respondent the following question: "Should women have complete freedom to decide whether to have an abortion? Yes or No?" Many Canadians feel that abortion is justified only in certain situations, and so would want to answer "it depends" to such a generic question, but instead, are led to believe they have to force-fit their answer into a yes/no category. The interviewers do record "in between" or "don't know" responses, but do not tell respondents up-front that these are valid options. So many respondents will try to answer simply "yes" or "no." To do this, they have to interpret what the interviewer is asking. Is the interviewer asking if the choice to abort in any and all circumstances should be a woman's right? For example, does she have the right to use abortion as a method of birth control? Or is the interviewer asking if a woman should be free from coercion to abort by, say, her boyfriend or a parent? Or is the interviewer asking if the decision to abort in the case of rape or in the case when the woman's life or health is at stake should be left up to the woman as opposed to her boyfriend or her doctor? Each respondent can interpret the same question differently.

A more accurate assessment of people's views on abortion could be achieved by allowing respondents to answer "it depends" and specifying what it depends on (e.g., rape, risk to life of mother, age of fetus, etc.) -similar to how Gallup has conducted polls in the past and a recent Léger poll. When asked this way, polls consistently show support for unrestricted abortion at less than a third.

With regards to this question being a leading question, the purpose of the poll was to find out people's opinions on five freedoms: economic, religion, association, speech, and body. "Freedom" is good and almost everyone in our society would agree with that. Freedom of religion, of speech, of association-all good things, right?

Basically, "freedom" is a loaded term. It has all kinds of connotations attached to it and they are good ones. Then just before the abortion question is asked, the interviewer says, "Turning to freedom of the body, should women have complete freedom to decide whether to have an abortion? Yes or No?" The respondent has been set up and can be predisposed to answering "yes" simply because the question is posed in the context of freedom.

A related problem is one of bias. The problem with asking the abortion question under the category "freedom of the body" is that it frames abortion as an issue of bodily freedom. But that is one of the basic points of controversy in the whole abortion debate in the first place: one side says it is an issue of a woman's right to do what she wants with her own body; the other side says the fetus is a unique human being and, although in the woman's body, is not part of the woman's body, so a woman can do what she wants with her own body but not with the body of the fetus.

So given that this is at the crux of the controversy in the first place, when the interviewer prefixes the question with "turning to freedom of the body," the interviewer has basically chosen sides in the controversy (by stating that abortion is an issue of bodily freedom). This can lead to a false answer. That is, the respondent might answer based on what he thinks the interviewer wants to hear, or on what he perceives is normative, rather than on what he truly thinks. If he believes (because of the way the interviewer asked the question) that society accepts the belief that abortion is an issue of bodily freedom-in other words, that this belief is a societal norm-then the respondent may be biased towards giving what he deems to be the most socially acceptable answer-something known an "social-desirability bias."

Because of the ambiguous, leading, and biased nature of the abortion question, respondents were not necessarily "free" to answer how they truly felt about the issue. Thus the results of the COMPAS/National Post poll do not give a valid reflection of Canadians' true feelings about abortion.

A disservice is done to the public, to politicians, to lawmakers, and, hence, to all Canadians when such results are reported as if they were valid. The laws a society enacts are often (not always) based on what the public wants. And polls are one means of finding out what that is. They help to inform the decisions made by those who run the country. And such important decisions should not be made based on misinformation.- BM