A Validation of R-Indicators as a Measure of the Risk of Bias using Data from a Nonresponse Follow-Up Survey

AAPOR, American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition.Search in Google Scholar

Bethlehem, J.G. 2002. “Weighting Non-response Adjustment Based on Auxiliary Information.” In Survey Non-response, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.Search in Google Scholar

Beullens, K. and G. Loosveldt. 2012. “Should High response Rates Really be the Primary Objective?” Survey Practice 5(3): 1–5. DOI: https://doi.org/10.29115/SP-2012-0019.10.29115/SP-2012-0019Search in Google Scholar

Beullens, K., G. Loosveldt, C. Vandenplas, and I. Stoop. 2018. “Response Rates in the European Social Survey: Increasing, Decreasing, or a Matter of Fieldwork Efforts?” Survey Methods: Insights from the Field. DOI: https://doi.org/10.13094/SMIF-2018-00003.Search in Google Scholar

Brick, J.M. and M.E. Jones. 2008. “Propensity to respond and nonresponse bias.” METRON – International Journal of Statistics, LXVI(1), 51–73.Search in Google Scholar

Brick, J.M. and D. Williams. 2013. “Explaining rising Non-response Rates in Cross-sectional Surveys.” The ANNALS of the American Academy of Political and Social Science 645: 36–59. DOI: https://doi.org/10.1177/0002716212456834.10.1177/0002716212456834Search in Google Scholar

Brick, J.M. and R. Tourangeau. 2017. “Responsive Survey Designs for Reducing Non-response Bias.” Journal of Official Statistics 33(3): 735–752. DOI: http://dx.doi.org/10.1515/JOS-2017-0034.10.1515/jos-2017-0034Search in Google Scholar

Cobben, F. 2009. Non-response in Sample Surveys. Methods for Analysis and Adjustment. PhD thesis, The Hague: Statistics Netherlands. Available at: https://hdl.handle.-net/11245/1.312964 (accessed June 2020).Search in Google Scholar

Cornesse, C. and M. Bosnjak. 2018. “Is there an association between survey characteristics and representativeness? A meta-analysis.” Survey Research Methods 12(1): 1–13. DOI: https://doi.org/10.18148/srm/2018.v12i1.7205.Search in Google Scholar

De Heij, V., B. Schouten, and N. Shlomo. 2015. RISQ manual 2.1. Tools in SAS and R for the computation of R-indicators, partial R-indicators and partial coefficients of variation. RISQ Project. Available at: www.risq-project.eu.Search in Google Scholar

De Leeuw, E. and W. de Heer. 2002. “Trends in Household Survey Non-response: A Longitudinal and International Comparison.” In Survey Non-response, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.Search in Google Scholar

Ernst Stähli, M., D. Joye, M. Sapin, A. Pollien, M. Ochsner, and A. van den Hende. 2018. “Non Response Surveys (NRS): ESS 2006, EVS 2008, ESS 2010, MOSAiCH 2011, ESS 2012, ESS 2014.” [Dataset]. Distributed by FORS, Lausanne. https://doi.org/10.23662/FORS-DS-697-1.Search in Google Scholar

ESS Round 5: European Social Survey 2016: ESS-5 2010 Documentation Report. Edition 4.1. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/docs/round5/survey/ESS5_data_documentation_report_e04_2.pdf (accessed June 2020).Search in Google Scholar

ESS Round 5: European Social Survey Round 5 Data. 2010. Data file edition 3.4. NSD – Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/download.html?file=ESS5CH&c=CH&y=2010.Search in Google Scholar

Groves, R.M. 2006. “Non-response Rates and Non-response Bias in Household surveys.” Public Opinion Quarterly 70, Special Issue: 646–675. DOI: https://doi.org/10.1093/poq/nfl033.10.1093/poq/nfl033Search in Google Scholar

Groves, R.M. and S. Heeringa. 2006. “Responsive design for household surveys: tools for actively controlling survey errors and costs.” Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3): 439–457. DOI: https://doi.org/10.1111/j.1467-985X.2006.00423.x10.1111/j.1467-985X.2006.00423.xSearch in Google Scholar

Groves, R.M. and E. Peytcheva. 2008. “The Impact of Non-response Rates on Non-response Bias – A Meta-Analysis.” Public Opinion Quarterly 72: 167–189. DOI: https://doi.org/10.1093/poq/nfn011.10.1093/poq/nfn011Search in Google Scholar

Groves, R.M. and M.P. Couper. 1998. Non-Response in Household Interview Survey. New York: John Wiley & Sons.10.1002/9781118490082Search in Google Scholar

Groves, R.M., J.M. Brick, M.P. Couper, W. Kalsbeek, B. Harris-Kojetin, F. Kreuter, B.-E. Pennell, T. Raghunathan, B. Schouten, T. Smith, R. Tourangeau, A. Bowers, M. Jans, C. Kennedy, R. Levenstein, K. Olson, E. Peytcheva, S. Ziniel, and J. Wagner. 2008. “Issues Facing the Field: Alternative Practical Measures of Representativeness of Survey Respondent Pools.” Survey Practice 1(3): 1–6. DOI: https://doi.org/10.29115/SP-2008-0013.10.29115/SP-2008-0013Search in Google Scholar

Gummer, T. and J.E. Blummenstiel. 2018. “Experimental Evidence on Reducing Nonresponse Bias through Case Prioritization: The Allocation of Interviewers.” Field Methods 30(2): 124–139. DOI: https://doi.org/10.1177/1525822X18757967.10.1177/1525822X18757967Search in Google Scholar

Kreuter, F. 2013. “Facing the Nonresponse Challenge.” The Annals of the American Academy of Political and Social Science 645(1): 23–35. DOI: https://doi.org/10.1177/0002716212456815.10.1177/0002716212456815Search in Google Scholar

Kreuter, F. and K. Olson. 2011. “Multiple Auxiliary Variables in Nonresponse Adjustment.” Sociological Methods & Research 40(2): 311–332. DOI: https://doi.org/10.1177/0049124111400042.10.1177/0049124111400042Search in Google Scholar

Lin, I.-F. and N.C. Schaeffer. 1995. “Using Survey Participants to Estimate the Impact of Nonparticipation.” Public Opinion Quarterly 59(2): 236–258, DOI: https://doi.org/10.1086/269471.10.1086/269471Search in Google Scholar

Little, R.J.A. and S. Vartivarian. 2005. “Does Weighting for Non-Response increase the Variance of Survey Means?” Survey Methodology 31(2): 161–68. DOI: https://doi.org/10.2307/1403140.10.2307/1403140Search in Google Scholar

Little, R.J.A. 1986. “Survey Nonresponse Adjustments for Estimates of Means.” International Statistical Review / Revue Internationale de Statistique 54(2), 139–157. JSTOR. DOI: https://doi.org/10.2307/1403140.10.2307/1403140Search in Google Scholar

Little, R.J.A. and D.B. Rubin. 2014. Statistical Analysis with Missing Data. London, UK: John Wiley & Sons.Search in Google Scholar

Luiten, A. and B. Schouten, 2013. “Tailored Fieldwork Design to increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 176(1): 169–189. DOI: https://doi.org/10.1111/j.1467-985X.2012.01080.x.10.1111/j.1467-985X.2012.01080.xSearch in Google Scholar

Matsuo, H., J. Billiet, G. Loosveldt, F. Berglund, and Ø. Kleven. 2010. “Measurement and Adjustment of Non-response Bias based on Non-response Survey: the Case of Belgium and Norway in the European Social Survey Round 3.” Survey Research Methods 4(3): 165–178. DOI: http://dx.doi.org/10.18148/srm/2010.v4i3.3774.Search in Google Scholar

McCaffrey, D.F., G. Ridgeway, and A.R. Morral. 2004. “Propensity Score Estimation with Boosted Regression for Evaluating Causal Effects in Observational Studies.” Psychological Methods 9: 403–425. DOI: https://doi.org/10.1037/1082-989X.9.4.403.10.1037/1082-989X.9.4.40315598095Search in Google Scholar

Moore, J.C., G.B. Durrant, and P.W. Smith. 2018. “Data Set Representativeness during Data Collection in Three UK Social Surveys: Generalizability and the Effects of Auxiliary Covariate Choice.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 181(1): 229–248. DOI: https://doi.org/10.1111/rssa.12256.10.1111/rssa.12256Search in Google Scholar

Nagelkerke, N.J. 1991. “A Note on a General Definition of the Coefficient of Determination.” Biometrika 78(3): 691–692. DOI: https://doi.org/10.1093/biomet/78.3.691.10.1093/biomet/78.3.691Search in Google Scholar

Nishimura, R., J. Wagner, and M. Elliott. 2016. “Alternative Indicators for the Risk of Non-response Bias: A Simulation Study.” International Statistical Review 84(1): 43–62. DOI: https://doi.org/10.1111/insr.12100.10.1111/insr.12100487131627212786Search in Google Scholar

Olmos, A. and P. Govindsamy. 2015. “A Practical Guide for using Propensity Score Weighting in R.” Practical Assessment, Research & Evaluation, 20(13). Available at: http://pareonline.net/getvn.asp?v=20&n=13. (accessed June 2020).Search in Google Scholar

Olson, K. 2006. “Survey participation, Non-Response Bias, Measurement Error Bias, and Total Bias.” Public Opinion Quarterly 70(5): 737–758. DOI: https://doi.org/10.1093/-poq/nfl038.Search in Google Scholar

Olson, K. 2013. “Do Non-response Follow-ups Improve or Reduce Data Quality?: A Review of the Existing Literature.” Journal of the Royal Statistical Society: Series A: Statistics in Society 176(1): 129–145. DOI: http://doi.org/10.1111/j.1467-985X.2012.01042.x.10.1111/j.1467-985X.2012.01042.xSearch in Google Scholar

Peytchev, A., E. Peytcheva, and R.M. Groves. 2010. “Measurement Error, Unit Nonresponse, and Self-Reports of Abortion Experiences.” Public Opinion Quarterly 74: 319–327. DOI: https://doi.org/10.1093/poq/nfq002.10.1093/poq/nfq002Search in Google Scholar

Peytcheva, E. and R.M. Groves. 2009. “Using Variation in Response Rates of Demographic Subgroups as Evidence of Non-response Bias in Survey Estimates.” Journal of Official Statistics 25: 193–201. Available at: https://www.researchgate.net/publication/282119961_Using_Variation_in_Response_Rates_of_Demographic_Subgroups_as_Evidence_of_Nonresponse_Bias_in_Survey_Estimates (accessed June 2020).Search in Google Scholar

Roberts, C. and C. Vandenplas. 2017. “Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes.” Journal of Official Statistics 33(2): 303–334. DOI: https://doi.org/10.1515/jos-2017-0016.10.1515/jos-2017-0016Search in Google Scholar

Roberts, C., C. Vandenplas, and M. Ernst Stähli. 2014a. “Using Register Data to assess the Impact of Response Enhancement Methods on the Risk of Non-response Bias.” Survey Research Methods 8(2): 67–80. DOI: http://dx.doi.org/10.18148/srm/2014.v8i2.5459.Search in Google Scholar

Roberts, C., N. Allum, and P. Sturgis. 2014b. “Non-response and Measurement error in an online panel: Does additional Effort to Recruit Reluctant Respondents Result in Poorer Quality Data?” In Online Panel Research: A Data Quality Perspective. edited by M. Callegaro, R. Baker, J. Bethlehem, A.S. Göritz, J.A. Krosnick, and P.J. Lavrakas. Hoboken: Wiley, Survey Methodology Series.Search in Google Scholar

Sakshaug, J.F. and M. Antoni. 2018. “Evaluating the Utility of Indirectly Linked Federal Administrative Records for Nonresponse Bias Adjustment.” Journal of Survey Statistics and Methodology 7(2): 227–249. DOI: https://doi.org/10.1093/jssam/smy009.10.1093/jssam/smy009Search in Google Scholar

Särndal, C.-E. and S. Lundström. 2010. “Design for Estimation: Identifying Auxiliary Vectors to Reduce Nonresponse Bias.” Survey Methodology 36(2): 131–144.Search in Google Scholar

Schouten, B. 2018. “Statistical inference based on randomly generated auxiliary variables.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 80(1): 33–56. DOI: https://doi.org/10.1111/rssb.12242.10.1111/rssb.12242Search in Google Scholar

Schouten, B. and N. Shlomo. 2017. “Selecting Adaptive Survey Design Strata with Partial R-indicators.” International Statistical Review 85(1): 143–163. DOI: https://doi.org/10.1111/insr.12159.10.1111/insr.12159Search in Google Scholar

Schouten, B. and F. Cobben. 2007. “R-Indexes for the Comparison of Different Fieldwork Strategies and Data Collection Modes.” Statistics Netherlands. Discussion Paper, 07002, CBC, Voorburg. Available at: http://hummedia.manchester.ac.uk/institutes/cmist/risq/schouten-cobben-2007-a.pdf (accessed June 2020).Search in Google Scholar

Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology 35(1): 101–113. Available at: https://pdfs.semanticscholar.org/aa59/4bf03a7cc219ccc6da01d1e3cb14a125d67a.pdf (accessed June 2020).Search in Google Scholar

Schouten, B., F. Cobben, P. Lundquist, and J. Wagner. 2016. “Does more Balanced Response imply less Non-response Bias?” Journal of the Royal Statistical Society, Series A (Statistics in Society) 179(3): 727–748. DOI: https://doi.org/10.1111/rssa.12152.10.1111/rssa.12152Search in Google Scholar

Schouten, B., J. Bethlehem, K. Beullens, Ø. Kleven, G. Loosveldt, A. Luiten, K. Rutar, N. Shlomo, and C. Skinner. 2012. “Evaluating, Comparing, Monitoring, and Improving Representativeness of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80(3): 382–399. DOI: https://doi.org/10.1111/j.1751-5823.2012.00189.x.10.1111/j.1751-5823.2012.00189.xSearch in Google Scholar

Schouten, B., N. Shlomo, and C. Skinner. 2010. “Indicators for Representative Response.” Paper presented at Quality in Official Statistics Conference 2010, Helinski, Finland. Available at: https://q2010.stat.fi/ (accessed June 2020).Search in Google Scholar

Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativeness of Response.” Journal of Official Statistics 27(2): 231–253. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/indicators-for-monitoring-and-improving-representativeness-of-response.pdf (accessed June 2020).Search in Google Scholar

Stoop, I.A. 2004. “Surveying Nonrespondents.” Field Methods 16(1): 23–54. DOI: https://doi.org/10.1177/1525822X03259479.10.1177/1525822X03259479Search in Google Scholar

Stoop, I., J. Billiet, A. Koch, and R. Fitzerald. 2010. Improving Survey Response: Lessons Learned from the European Social Survey. London, UK: John Wiley and Sons Ltd.10.1002/9780470688335Search in Google Scholar

Tancreto, J.G. and M. Bentley. 2005. “Determining the Effectiveness of Multiple Non-response Follow-up Contact Attempts on Response and Data Quality.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2005. 3626–3632. Minneapolis, MN: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2005f.html (accessed June 2020).Search in Google Scholar

Vandenplas, C., D. Joye, M. Ernst Stähli, and A. Pollien. 2015. “Identifying Pertinent Variables for Nonresponse Follow-Up Surveys. Lessons Learned from 4 Cases in Switzerland.” Survey Research Methods 9(3): 141–158. DOI: https://doi.org/10.18148/srm/2015.v9i3.6138.Search in Google Scholar

Voogt, R.J.J. and W.E. Saris, 2005. “Mixed Mode Designs: Finding the Balance Between Non-response Bias and Mode Effects.” Journal of Official Statistics, 21(3): 367–387. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/-mixed-mode-designs-finding-the-balance-between-nonresponse-bias-and-mode-effects.pdf (accessed June 2020).Search in Google Scholar

Wagner, J. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly 76(3): 555–575. DOI: https://doi.org/10.1093/poq/nfs032.10.1093/poq/nfs032627697530538342Search in Google Scholar

Williams, D. and J.M. Brick. 2017. “Trends in U.S. Face-To-Face Household Survey Nonresponse and Level of Effort.” Journal of Survey Statistics and Methodology 6(2): 186–211. DOI: https://doi.org/10.1093/jssam/smx019.10.1093/jssam/smx019Search in Google Scholar

Yan, T., R. Tourangeau, and Z. Arens. 2004. “When less is more: Are reluctant respondents poor reporters?” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2004. 4632–4651. Toronto: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2004/files/Jsm2004-000169.pdf (accessed July 2020).Search in Google Scholar