Framing Effects in Survey Research: Consistency-Adjusted Estimators

Abstract

A well-known difficulty in survey research is that respondents may answer survey questions in ways that are systematically related to arbitrary features of the survey’s design, such as the way questions are worded or the order in which answers are presented. We develop a simple framework for analyzing such framing effects for binary survey questions in which individual survey-takers are observed under one of two frames. We first show that the conventional approach to analyzing data with framing effects – randomly assigning respondents across frames and then pooling the data – yields a biased estimate for the mean of the variable being measured. We then propose an alternative estimator, the consistency-adjusted mean, and provide conditions under which it identifies the mean of the survey variable for the subset of respondents whose answer is unaffected by the frame. When framing effects push respondents in heterogeneous directions, the consistency-adjusted mean will be biased, but less biased than the conventional approach. We also show how to estimate the distribution of covariates among the consistent and inconsistent respondents, and provide techniques for identifying the mean of the survey variable among the full population. We illustrate our proposed techniques by applying them to data from previous surveys characterized by framing effects.

Details

Author(s):
  • Jacob Goldin
  • Daniel Reck
Publish Date:
2016
Publication Title:
The American Statistician (forthcoming 2018)
Format:
Journal Article
Citation(s):
  • Jacob Goldin & Daniel Reck, Framing Effects in Survey Research: Consistency-Adjusted Estimators, February 25, 2015 (forthcoming in The American Statistician 2018).