Surveys and polls are invaluable tools for gauging public opinion on the critical issues of the day. As such, it is unsurprising that the mass media often rely on those tools for information. Unfortunately, surveys are inherently prone to mathematical errors that may distort the public opinion realities that they are trying to measure (Cox 1998). Stories that rely on them risk exacerbating this distortion by accepting such results uncritically. One common flaw in surveys is self-selection bias, which skews their results regarding demographics that are more likely to respond to them.
Since every survey can only reflect its respondents’ opinions, self-selection is a universal feature of surveys. It may be more useful to speak of self-selection bias as an incorrect assumption that a survey’s sample is truly random and representative (Cox 1998). In truth, the willingness to engage with a survey can depend on multiple factors, such as political beliefs or cultural taboos. Depending on how the survey is conducted, a lack of free time or Internet access can exclude potential respondents. Surveyors attempt to balance such factors through weighting based on demographical data, but this does not guarantee accurate representation.
A recent attempt to disprove the stereotype that the rural U.S. population is less concerned about the Covid-19 pandemic illustrates the possibility of self-selection bias. The article states that almost 9 out of 10 rural respondents plan to maintain social distancing, and only 5% believe the government mitigation measures are excessive (Marema 2020). The results have been obtained through an online poll conducted by Civis Analytics. The author acknowledges that the poll was based on self-selecting web panels, which inevitably excluded people with no Internet access (Marema 2020). What goes unmentioned is that this demographic is likely to be less informed about mitigation measures, affecting the results. Moreover, people with Internet access who take the pandemic less seriously would probably be less likely to respond. While Civis Analytics employed demographical weighting, it could not eliminate the problem.
The media often relies on surveys to illuminate the state of public opinion on various important issues. However, survey results are inevitably skewed by the self-selection of their participants. While weighting based on demographical data can help make surveys more representative of the population, it is a limited solution. Care must be taken to remember this likely source of distortion when examining the results of surveys and stories based on them.
References
Cox, Paul. 1998. “Glossary of Mathematical Mistakes.”
Marema, Tim. 2020. “Rural People Are Just as Worried about Covid-19 as Nation, Online Poll Says.” The Daily Yonder.