Responsive Survey Designs for Reducing Nonresponse Bias

Open access

Abstract

Survey researchers have been investigating alternative approaches to reduce data collection costs while mitigating the risk of nonresponse bias or to produce more accurate estimates within the same budget. Responsive or adaptive design has been suggested as one means for doing this. Falling survey response rates and the need to find effective ways of implementing responsive design has focused attention on the relationship between response rates and nonresponse bias. In our article, we re-examine the data compiled by Groves and Peytcheva (2008) in their influential article and show there is an important between-study component of variance in addition to the within-study variance highlighted in the original analysis. We also show that theory implies that raising response rates can help reduce the nonresponse bias on average across the estimates within a study. We then propose a typology of response propensity models that help explain the empirical findings, including the relative weak relationship between nonresponse rates and nonresponse bias. Using these results, we explore when responsive design tools such as switching modes, giving monetary incentives, and increasing the level of effort are likely to be effective. We conclude with some comments on the use of responsive design and weighting to control nonresponse bias.

Abraham, K., S. Helms, and S. Presser. 2009. “How Social Processes Distort Measurement: The Impact of Survey Nonresponse on Estimates of Volunteer Work in the United States.” American Journal of Sociology 114: 1129–1165.

Bartholomew, D.J. 1961. “A Method of Allowing for ‘Not-at-Home’ Bias in Sample Surveys.” Applied Statistics 10: 52–59. Doi: http://dx.doi.org/10.2307/2985408.

Bethlehem, J.G. 1988. “Reduction of Nonresponse Bias Through Regression Estimation.” Journal of Official Statistics 4: 251–260.

Brick, J.M. 2013. “Unit Nonresponse and Weighting Adjustments: A Critical Review.” Journal of Official Statistics 29: 329–353. Doi: https://doi.org/10.2478/jos-2013-0026.

Chun, Y. 2009. “Nonparticipation of the 12th graders in the National Assessment of Educational Progress: Understanding Determinants of Nonresponse and Assessing the Impact on NAEP Estimates of Nonresponse Bias According to Propensity Models.” University of Maryland. Available at: http://hdl.handle.net/1903/9916 (accessed June 4, 2017).

Curtin, R., S. Presser, and E. Singer. 2000. “The Effects of Response Rate Changes on the Index of Consumer Sentiment.” Public Opinion Quarterly 64: 413–428. Doi: https://doi.org/10.1086/318638.

Davern, M. 2013. “Nonresponse Rates are a Problematic Indicator of Nonresponse Bias in Survey Research.” Health Services Research 48: 905–912. Doi: http://dx.doi.org/10.1111/1475-6773.12070.

Groves, R. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70: 646–675. Available at: http://www.jstor.org/stable/4124220 (accessed July 9, 2017).

Groves, R.M., J.M. Brick, M. Couper, W. Kalsbeek, B. Harris-Kojetin, F. Kreuter, B.E. Pennell, T. Raghunathan, B. Schouten, T. Smith, and R. Tourangeau. 2008. “Issues Facing the Field: Alternative Practical Measures of Representativeness of Survey Respondent Pools.” Survey Practice 1: 14–22.

Groves, R.M. and M. Couper. 1998. Nonresponse in Household Interview Surveys. New York: John Wiley & Sons.

Groves, R.M., M.P. Couper, S. Presser, E. Singer, R. Tourangeau, G.P. Acosta, and L. Nelson. 2006. “Experiments in Producing Nonresponse Bias.” Public Opinion Quarterly 70: 720–736. Available at: http://www.jstor.org/stable/4124223 (accessed July 9, 2017).

Groves, R.M. and S.G. Heeringa. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 169: 439–457. Doi: http://dx.doi.org/10.1111/j.1467-985X.2006.00423.x.

Groves, R. and E. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias.” Public Opinion Quarterly 72: 167–189. Doi: https://doi.org/10.1093/poq/nfn011.

Groves, R.M., E. Singer, and A. Corning. 2000. “Leverage-Saliency Theory of Survey Participation: Description and an Illustration.” Public Opinion Quarterly 64: 299–308. Available at: http://www.jstor.org/stable/3078721 (accessed July 9, 2017).

Hansen, M.H. and W.N. Hurwitz. 1946. “The Problem of Nonresponse in Sample Surveys.” Journal of the American Statistical Association 41: 517–529. Doi: http://dx.doi.org/10.1080/01621459.1946.10501894.

Keeter, S., C. Kennedy, M. Dimock, J. Best, and P. Craighill. 2006. “Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey.” Public Opinion Quarterly 70: 759–779. Available at: http://www.jstor.org/stable/4124225 (accessed July 9, 2017).

Keeter, S., C. Miller, A. Kohut, R. Groves, and S. Presser. 2000. “Consequences of Reducing Nonresponse in a Large National Telephone Survey.” Public Opinion Quarterly 64: 125–148. Doi: https://doi.org/10.1086/317759.

Little, R.J.A. and S. Vartivarian. 2005. “Does Weighting for Nonresponse Increase the Variance of Survey Means?” Survey Methodology 31: 161–168.

Luiten, A. and B. Schouten. 2013. “Tailored Fieldwork Design to Increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society A 176: 169–189. Doi: http://dx.doi.org/10.1111/j.1467-985X.2012.01080.x.

Lundquist, P. and C.-E. Särndal. 2013. “Aspects of Responsive Design with Applications to the Swedish Living Conditions Survey.” Journal of Official Statistics 29: 557–582. Doi: https://doi.org/10.2478/jos-2013-0040.

McQuillan, G.M., T.M. Ezzati-Rice, A.B. Siller, W. Visscher, and P. Hurley. 1994. “Risk Behavior and Correlates of Risk for HIV Infection in the Dallas County Household HIV Survey.” American Journal of Public Health 84: 747–753. Doi: http://dx.doi.org/10.2105/AJPH.84.5.747.

Mercer, A., A. Caporaso, D. Cantor, and R. Townsend. 2015. “How Much Gets You How Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly 79: 105–129. Doi: https://doi.org/10.1093/poq/nfu059.

Merkle, D. and M. Edelman. 2002. “Nonresponse in Exit Polls: a Comprehensive Analysis.” In Survey Nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R. Little, 243–258. New York: John Wiley.

Messer, B.L. and D.A. Dillman. 2011. “Surveying the General Public Over the Internet Using Address-Based Sampling and Mail Contact Procedures.” Public Opinion Quarterly 75: 429–457. Doi: https://doi.org/10.1093/poq/nfr021.

Moore, J., G. Durrant, and P. Smith. 2016. “Data Set Representativeness during Data Collection in Three UK Social Surveys: Generalizability and the Effects of Auxiliary Covariate Choice.” Journal of the Royal Statistical Society, A. Doi: http://dx.doi.org/10.1111/rssa.12256.

Särndal, C.-E. 2011. “The 2010 Morris Hansen Lecture: Dealing with Survey Nonresponse in Data Collection, in Estimation.” Journal of Official Statistics 27: 1–21.

Särndal, C.-E., K. Lumiste, and I. Traat. 2016. “Reducing the Response Imbalance: Is the Accuracy of the Survey Estimates Improved?” Survey Methodology 42: 219–238.

Särndal, C.E. and S. Lundström. 2005. Estimation in Surveys with Nonresponse. New York: John Wiley & Sons.

Särndal, C.-E. and P. Lundquist. 2014. “Accuracy in Estimation with Nonresponse: a Function of the Degree of Imbalance and Degree of Explanation.” Journal of Survey Statistics and Methodology 2: 361–387. Doi: https://doi.org/10.1093/jssam/smu014.

Singer, E. 2002. “Use of Incentives to Reduce Nonresponse in Household Surveys.” In Survey Nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R. Little, 163–177. Wiley.

Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology 35: 101–113.

Schouten, B., F. Cobben, P. Lundquist, and J. Wagner. 2016. “Does More Balanced Survey Response Imply Less Non-Response bias?” Journal of the Royal Statistical Society, A 179: 727–748. Doi: http://dx.doi.org/10.1111/rssa.12152.

Silver, N. 2014. “Is the Polling Industry in Stasis or in Crisis?” Available at: https://fivethirtyeight.com/features/is-the-polling-industry-in-stasis-or-in-crisis/ (accessed February 22, 2016).

Sturgis, P., J. Williams, I. Brunton-Smith, and J. Moore. 2017. “Fieldwork Effort, Response Rate, and the Distribution of Survey Outcomes: A Multilevel Meta-Analysis.” Public Opinion Quarterly 81: 523–542. Doi: https://doi.org/10.1093/poq/nfw055.

Tourangeau, R., J.M. Brick, S. Lohr, and J. Li. 2017. “Adaptive and Responsive Survey Designs: a Review and Assessment.” Journal of the Royal Statistical Society A 180: 203–223. Doi: http://dx.doi.org/10.1111/rssa.12186.

Tourangeau, R., R.M. Groves, and C. Redline. 2010. “Sensitive Topics and Reluctant Respondents: Demonstrating a Link Between Nonresponse Bias and Measurement Error.” Public Opinion Quarterly 74: 413–432. Doi: https://doi.org/10.1093/poq/nfq004.

Trussell, N. and P. Lavrakas. 2004. “The Influence of Incremental Increases in Token Cash Incentives on Mail Survey Response.” Public Opinion Quarterly 68: 349–367. Doi: https://doi.org/10.1093/poq/nfh022.

U.S. Census Bureau. 2014. “American Community Survey Design and Methodology.” Available at: https://www.census.gov/programs-surveys/acs/methodology/design-and-methodology.html (accessed February 19, 2016).

Journal of Official Statistics

The Journal of Statistics Sweden

Journal Information


IMPACT FACTOR 2017: 0.662
5-year IMPACT FACTOR: 1.113

CiteScore 2017: 0.74

SCImago Journal Rank (SJR) 2017: 1.158
Source Normalized Impact per Paper (SNIP) 2017: 0.860

Cited By

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 695 695 65
PDF Downloads 344 344 31