Unit Nonresponse and Weighting Adjustments: A Critical Review

Open access

Abstract

This article reviews unit nonresponse in cross-sectional household surveys, the consequences of the nonresponse on the bias of the estimates, and methods of adjusting for it. We describe the development of models for nonresponse bias and their utility, with particular emphasis on the role of response propensity modeling and its assumptions. The article explores the close connection between data collection protocols, estimation strategies, and the resulting nonresponse bias in the estimates. We conclude with some comments on the current state of the art and the need for future developments that expand our understanding of the response phenomenon.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Andridge R.H. and Little R.J. (2011). Proxy Pattern-Mixture Analysis for Survey Nonresponse. Journal of Official Statistics 27 153-180.

  • Atrostic B.K. Bates N. Burt G. and Silberstein A. (2001). Nonresponse in U.S. Government Household Surveys: Consistent Measures Recent Trends and New Insights. Journal of Official Statistics 17 209-226.

  • Bartholomew D.J. (1961). A Method of Allowing for ‘Not-at-Home’ Bias in Sample Surveys. Applied Statistics 10 52-59.

  • Bates N. Dahlhamer J. and Singer E. (2008). Privacy Concerns too Busy or Just not Interested: Using Doorstep Concerns to Predict Survey Nonresponse. Journal of Official Statistics 24 591-612.

  • Beaumont J.F. (2005). On the Use of Data Collection Process Information for the Treatment of Unit Nonresponse Through Weight Adjustment. Survey Methodology 31 227-231.

  • Bethlehem J.G. (1988). Reduction of Nonresponse Bias Through Regression Estimation. Journal of Official Statistics 4 251-260.

  • Bethlehem J.G. (2002). Weighting Nonresponse Adjustments Based on Auxiliary Information. Survey Nonresponse R.M. Groves D.A. Dillman J.L. Eltinge and R.J.A. Little (eds). New York: Wiley.

  • Bethlehem J. Cobben F. and Schouten B. (2011). Handbook in Nonresponse in Household Surveys. New York: Wiley.

  • Brehm J. (1993). The Phantom Respondents: Opinion Surveys and Political Representation. Ann Arbor: University of Michigan Press.

  • Brick J.M. and Jones M.E. (2008). Propensity to Respond and Nonresponse Bias. Metron-International Journal of Statistics LXVI 51-73.

  • Brick J.M. and Kalton G. (1996). Handling Missing Data in Survey Research. Statistical Methods in Medical Research 5 215-238.

  • Brick J.M. and Montaquila J.M. (2009). Nonresponse and Weighting. Handbook of Statistics. Sample Surveys: Design Methods and Applications D. Pfeffermann and C.R. Rao (eds). Vol. 29A. Amsterdam: Elsevier-North Holland 163-186.

  • Brick J.M. Montaquila J. Han D. and Williams D. (2012). Improving Response Rates for Spanish-Speakers in Two-Phase Mail Surveys. Public Opinion Quarterly 76 721-732.

  • Brick J.M. and Williams D. (2013). Explaining Rising Nonresponse Rates in Cross- Sectional Surveys. The ANNALS of the American Academy of Political and Social Science 645 36-59.

  • Cassel C. Särndal C.-E. and Wretman J. (1983). Some Uses of Statistical Models in Connection With the Nonresponse Problem. Incomplete Data in Sample Surveys W.G. Madow and I. Olkin (eds). Vol. 3. New York: Academic Press.

  • Chang T. and Kott P.S. (2008). Using Calibration Weighting to Adjust for Nonresponse Under a Plausible Model. Biometrika 95 557-571.

  • Cochran W. (1977). Sampling Techniques (3rd edition). New York: Wiley.

  • Colley R.H. (1945). Don’t Look Down Your Nose at Mail Questionnaires. Printers’ Ink March 16 21-108.

  • Curtin R. Presser S. and Singer E. (2000). The Effects of Response Rate Changes on the Index of Consumer Sentiment. Public Opinion Quarterly 64 413-428.

  • Curtin R. Presser S. and Singer E. (2005). Changes in Telephone Survey Nonresponse Error Over the Past Quarter Century. Public Opinion Quarterly 69 87-98.

  • Da Silva D.N. and Opsomer J.D. (2004). Properties of the Weighting Cell Estimator Under a Nonparametric Response Mechanism. Survey Methodology 30 45-55.

  • Da Silva D.N. and Opsomer J.D. (2009). Nonparametric Propensity Weighting for Survey Nonresponse Through Local Polynomial Regression. Survey Methodology 35 165-176.

  • Dalenius T. (1983). Some Reflections on the Problem of Missing Data. Incomplete Data in Sample Surveys W.G. Madow and I. Olkin (eds). Vol. 3. New York: Academic Press 411-413.

  • David M. Little R. Samuhel M. and Triest R. (1983). Nonrandom Nonresponse Models Based on the Propensity to Respond. Proceedings of the Business and Economic Statistics Section of the American Statistical Association 168-173.

  • David M. Little R.J.A. Samuhel M. and Triest R. (1986). Alternative Methods for CPS Income Imputation. Journal of the American Statistical Association 81 29-41.

  • De Leeuw E. and De Heer W. (2002). Trends in Household Survey Nonresponse: A Longitudinal and International Comparison. Survey Nonresponse R.M. Groves D.A.

  • Dillman J.L. Eltinge and R.J.A. Little (eds). New York: Wiley 41-54.

  • Deming W. (1953). On a Probability Mechanism to Attain an Economic Balance Between Resultant Error of Response and the Bias of Nonresponse. Journal of the American Statistical Association 48 743-772.

  • Deville J.C. and Särndal C.-E. (1992). Calibration Estimators in Survey Sampling. Journal of the American Statistical Association 87 376-382.

  • Dillman D. (1978). Mail and Telephone Surveys: The Total Design Method. New York: Wiley.

  • Dillman D. Smyth J. and Christian L. (2009). Internet Mail and Mixed-Mode Surveys: The Tailored Design Method (3rd edition). New York: Wiley.

  • Dunkelburg W. and Day G. (1973). Nonresponse Bias and Callbacks in Sample Surveys. Journal of Marketing Research 10 160-168.

  • Ferber R. (1949). The Problem of Bias in Mail Returns: A Solution. Public Opinion Quarterly 12 669-676.

  • Feskins R. Hoop J. Lensvelt-Mulders G. and Schmeets H. (2011). Collecting Data Among Ethnic Minorities in an International Perspective. Field Methods 18 284-304.

  • Fuller W.A. Loughin M.M. and Baker H.D. (1994). Regression Weighting for the 1987-88 National Food Consumption Survey. Survey Methodology 20 75-85.

  • Goyder J. (1987). The Silent Minority: Nonrespondents on Sample Surveys. Boulder CO: Westview Press.

  • Greenlees J. Reece W. and Zieschang K. (1982). Imputation of Missing Values When the Probability of Response Depends on the Variable Being Imputed. Journal of the American Statistical Association 77 251-261.

  • Groves R.M. (2006). Nonresponse Rates and Nonresponse Bias in Household Surveys.

  • Groves R.M. and Couper M.P. (1998). Nonresponse in Household Interview Surveys. New York: Wiley.

  • Groves R.M. Couper M. Presser S. Singer E. Tourangeau R. Acosta G.P. and Nelson L. (2006). Experiments in Producing Nonresponse Bias. Public Opinion Quarterly 70 720-736.

  • Groves R. Dillman D. Eltinge J. and Little R. (2002). Survey Nonresponse. New York: Wiley 41-54.

  • Groves R.M. and Heeringa S.G. (2006). Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs. Journal of the Royal Statistical Society Series A 169 439-457.

  • Hansen M.H. and Hurwitz W.N. (1946). The Problem of Non-Response in Sample Surveys. Journal of the American Statistical Association 41 517-529.

  • Haring R. Alte D. Völzkea H. Sauer S. Wallaschofski H. John U. and Schmidt C. (2009). Extended Recruitment Efforts Minimize Attrition but not Necessarily Bias. Journal of Clinical Epidemiology 62 252-260.

  • Hartley H.O. (1946). Discussion of “A Review of Recent Statistical Developments in Sampling and Sample surveys.”. Journal of the Royal Statistical Society 109 37-38.

  • Heckman J. (1979). Sample Selection Bias as a Specification Error. Econometrica 47 153-162.

  • Holt D. and Smith T.M.F. (1979). Post-Stratification. Journal of the Royal Statistical Society Series A 142 33-46.

  • Ingen E. Stoop I. and Breedveld K. (2009). Nonresponse in the Dutch Time Use Survey: Strategies for Response Enhancement and Bias Reduction. Field Methods 21 69-90.

  • Kalton G. (1983). Compensating for Missing Survey Data. Ann Arbor: University of Michigan Press.

  • Kalton G. and Flores-Cervantes I. (2003). Weighting Methods. Journal of Official Statistics 18 81-97.

  • Kalton G. and Kasprzyk D. (1986). The Treatment of Missing Survey Data. Survey Methodology 12 1-16.

  • Keeter S. Miller C. Kohut A. Groves R.M. and Presser S. (2000). Consequences of Reducing Nonresponse in a Large National Telephone Survey. Public Opinion Quarterly 64 125-148.

  • Kreuter F.OlsonK.Wagner J. YanT.Ezzati-Rice T.M.Casas-Cordero C. LemayM. Peytchev A. Groves R.M. and Raghunathan T.E. (2010). Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Non-Response: Examples from Multiple Surveys. Journal of the Royal Statistical Society Series A 173 389-407.

  • Lin I.-F. and Schaeffer N.C. (1995). Using Survey Participants to Estimate the Impact of Nonparticipation. Public Opinion Quarterly 59 236-258.

  • Little R.J.A. (1986). Survey Nonresponse Adjustments for Estimates of Means. International Statistical Review 54 139-157.

  • Little R.J.A. (1993). Pattern-Mixture Models for Multivariate Incomplete Data. Journal of the American Statistical Association 88 125-134.

  • Little R.J.A. and Rubin D.B. (2002). Statistical Analysis With Missing data (2nd edition). New York: Wiley.

  • Lumley T. Shaw P. and Dai J. (2011). Connections Between Survey Calibration Estimators and Semiparametric Models for Incomplete Data. International Statistical Review 79 200-220.

  • Lundstro¨m S. and Särndal C.-E. (1999). Calibration as a Standard Method for Treatment of Nonresponse. Journal of Official Statistics 15 305-327.

  • Madow W.G. Nisselson H. and Olkin I. (1983). Incomplete Data in Sample Surveys Vol. 1. New York: Academic Press.

  • Madow W.G. and Olkin I. (1983). Incomplete Data in Sample Surveys Vol. 3. New York: Academic Press. Madow W.G. Olkin I. and Rubin D.B. (1983). Incomplete Data in Sample Surveys Vol. 2. New York: Academic Press.

  • Merkle D. Edelman M. Dykeman K. and Brogan C. (1998). An Experimental Study of Ways to Increase Exit Poll Response Rates and Reduce Survey Error. Paper presented at the Annual Conference of the American Association for Public Opinion Research St. Louis MO.

  • Micklewright J. Schnepf S. and Skinner C. (2012). Non-Response Biases in Surveys of Schoolchildren: The Case of the English Programme for International Student Assessment (PISA) samples. Journal of the Royal Statistical Society Series A 175 915-938.

  • Mohadjer L. Berlin M. Rieger S. Waksberg J. Rock D. Yamamoto K. Kirsch I. and Kolstad A. (1997). The Role of Incentives in Literacy Survey Research. Adult Basic Skills: Innovations in Measurement and Policy Analysis A. Tuijnman I. Kirsch and D. Wagner (eds). Creskill NJ: Hampton Press.

  • Molenberghs G. Beunckens C. Sotto C. and Kenward M.G. (2008). Every Missingness not at Random Model has a Missingness at Random Counterpart With Equal Fit. Journal of the Royal Statistical Society: Series B 70 371-388.

  • Oh H.L. and Scheuren F.J. (1983). Weighting Adjustments for Unit Nonresponse. Incomplete Data in Sample Surveys W.G. Madow I. Olkin and D.B. Rubin (eds). Vol. 2. New York: Academic Press 143-184.

  • Olsen K. and Groves R.M. (2012). An Examination of Within-Person Variation in Response Propensity over the Data Collection Field Period. Journal of Official Statistics 28 29-51.

  • Peytcheva E. and Groves R.M. (2009). Using Variation in Response Rates of Demographic Subgroups as Evidence of Nonresponse Bias in Survey Estimates. Journal of Official Statistics 25 193-201.

  • Phipps P. and Toth D. (2012). Analyzing Establishment Nonresponse Using an Interpretable Regression Tree Model with Linked Administrative Data. Annals of Applied Statistics 6 772-794.

  • Politz A. and Simmons W. (1949). An Attempt to Get “Not at Homes” Into the Sample Without Callbacks. Journal of the American Statistical Association 44 9-31.

  • Rosenbaum P.R. and Rubin D.B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika 70 41-55.

  • Rubin D.B. (1976). Inference and Missing Data (with discussion). Biometrika 63 581-592.

  • Särndal C.-E. (2011a). Morris Hansen Lecture: Dealing With Survey Nonresponse in Data Collection in Estimation. Journal of Official Statistics 27 1-21.

  • Särndal C.-E. (2011b). Three Factors to Signal Non-Response Bias with Applications to Categorical Auxiliary Variables. International Statistical Review 79 233-254.

  • Särndal C.-E. and Lundstro¨m S. (2005). Estimation in Surveys with Nonresponse.Chichester UK: Wiley.

  • Särndal C.-E. and Lundstro¨m S. (2008). Assessing Auxiliary Vectors for Control of Nonresponse Bias in the Calibration Estimator. Journal of Official Statistics 4 251-260.

  • Särndal C.-E. and Lundstro¨m S. (2010). Design for Estimation: Identifying Auxiliary vectors to reduce nonresponse bias. Survey Methodology 36 131-144.

  • Särndal C.-E. Swensson B. and Wretman J. (1992). Model Assisted Survey Sampling. New York: Springer-Verlag.

  • Schmeets H. (2010). Increasing Response Rates and the Consequences in the Dutch Parliamentary Election Study 2006. Field Methods 22 391-412.

  • Schouten B. (2007). A Selection Strategy for Weighting Variables Under a Not-Missingat- Random Assumption. Journal of Official Statistics 23 51-68.

  • Schouten B. Calinescu M. and Luiten A. (2011a). Optimizing Quality of Response Through Adaptive Survey Designs. The Hague: Statistics Netherlands Available at: http://www.cbs.nl/NR/rdonlyres/2D62BF4A-6783-4AC4-8E4512EF20C6675C/0/2011x1018.pdf. (Accessed May 24 2013).

  • Schouten B. Cobben F. and Bethlehem J. (2009). Measures for the Representativeness of Survey Response. Survey Methodology 35 101-113.

  • Schouten B. Schlomo N. and Skinner C. (2011b). Indicators for Monitoring and Improving Representativeness of Response. Journal of Official Statistics 27 231-253.

  • Singer E. (2002). Use of Incentives to Reduce Nonresponse in Household Surveys.

  • Survey Nonresponse R. Groves D. Dillman J. Eltinge and R. Little (eds). New York: Wiley 163-177.

  • Singer E. and Ye C. (2013). The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science 645 112-141.

  • Skinner C.J. and D’Arrigo J. (2011). Inverse Probability Weighting for Clustered Nonresponse. Biometrika 98 953-966.

  • Smith T.W. (1995). Trends in Non-Response Rates. International Journal of Public Opinion Research 7 157-171.

  • Steeh C. Kirgis N. Cannon B. and DeWitt J. (2001). Are They Really as Bad as They Seem? Nonresponse Rates at the End of the Twentieth Century. Journal of Official Statistics 17 227-247.

  • Steele F. and Durrant G.B. (2011). Alternative Approaches to Multilevel Modelling of Survey Non-Contact and Refusal. International Statistical Review 79 70-91.

  • Stoop I.A.L. (2005). The Hunt for the Last Respondent: Nonresponse in Sample Surveys. The Hague: Social and Cultural Planning Office.

  • Stoop I. Billiet J. Koch A. and Fitzgerald R. (2010). Improving Survey Response: Lessons Learned from the European Social Survey. Chichester: Wiley.

  • Synodinos N.E. and Yamada S. (2000). Response Rate Trends in Japanese Surveys. International Journal of Public Opinion Research 12 48-72.

  • Tanur J. (1999). Looking Backwards and Forwards at the CASM Movement. Cognition and Survey Research M. Sirken D. Hermann S. Schechter N. Schwarz J. Tanur and R. Tourangeau (eds). New York: Wiley 13-20.

  • Thomsen I. (1973). A Note on the Efficiency of Weighting Subclass Means to Reduce the Effects of Nonresponse When Analyzing Survey Data. Statistisk Tidskrift 11 278-285.

  • Tourangeau R. Rips L.J. and Rasinski K. (2000). The Psychology of Survey Response. New York: Cambridge University Press. Wagner J. (2010). The Fraction of Missing Information as a Tool for Monitoring the Quality of Survey Data. Public Opinion Quarterly 74 223-243.

  • Wetzels W. Schmeets H. Van den Brakel J. and Feskens R. (2008). Impact of Prepaid Incentives in Face-to-Face Surveys: A Large-Scale Experiment With Postage Stamps. International Journal of Public Opinion Research 20 507-516.

  • Yates F. (1946). A Review of Recent Statistical Developments in Sampling and Sample Surveys. Journal of the Royal Statistical Society 109 12-43.

Search
Journal information
Impact Factor


IMPACT FACTOR 2018: 0,837
5-year IMPACT FACTOR: 0,934

CiteScore 2018: 1.04

SCImago Journal Rank (SJR) 2018: 0.963
Source Normalized Impact per Paper (SNIP) 2018: 1.020

Cited By
Metrics
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1173 739 36
PDF Downloads 599 426 12