Total Survey Error and Respondent Driven Sampling: Focus on Nonresponse and Measurement Errors in the Recruitment Process and the Network Size Reports and Implications for Inferences

Sunghee Lee 1 , Tuba Suzer-Gurtekin 1 , James Wagner 1 , and Richard Valliant 1
  • 1 Institute for Social Research, University of Michigan, 48104, Ann Arbor, United States of America


This study attempted to integrate key assumptions in Respondent-Driven Sampling (RDS) into the Total Survey Error (TSE) perspectives and examine TSE as a new framework for a systematic assessment of RDS errors. Using two publicly available data sets on HIV-at-risk persons, nonresponse error in the RDS recruitment process and measurement error in network size reports were examined. On nonresponse, the ascertained partial nonresponse rate was high, and a substantial proportion of recruitment chains died early. Moreover, nonresponse occurred systematically: recruiters with lower income and higher health risks generated more recruits; and peers of closer relationships were more likely to accept recruitment coupons. This suggests a lack of randomness in the recruitment process, also shown through sizable intra-chain correlation. Self-reported network sizes suggested measurement error, given their wide dispersion and unreasonable reports. This measurement error has further implications for the current RDS estimators, which use network sizes as an adjustment factor on the assumption of a positive relationship between network sizes and selection probabilities in recruitment. The adjustment resulted in nontrivial unequal weighting effects and changed estimates in directions that were difficult to explain and, at times, illogical. Moreover, recruiters’ network size played no role in actual recruitment. TSE may serve as a tool for evaluating errors in RDS, which further informs study design decisions and inference approaches.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Baker, R., J.M. Brick, N.A. Bates, M. Battaglia, M.P. Couper, J.A. Dever, K.J. Gile, and R. Tourangeau. 2013. “Summary Report of the AAPOR Task Force on Non-Probability Sampling.” Journal of Survey Statistics and Methodology 1(2): 90–143. Doi:

  • Beaumont, J.-F. 2008. “A New Approach to Weighting and Inference in Sample Surveys.” Biometrika 95(3): 539–553. Doi:

  • Burt, R.D., H. Hagan, K. Sabin, and H. Thiede. 2010. “Evaluating Respondent-Driven Sampling in a Major Metropolitan Area: Comparing Injection Drug Users in the 2005 Seattle Area National HIV Behavioral Surveillance System Survey with Participants in the RACEN and Kiwi Studies.” Annals of Epidemiology 20(2): 159–167. Doi:

  • Centers for Disease Control and Prevention (CDC). 2009. HIV-Associated Behaviors Among Injecting-Drug Users––23 Cities, United States, May 2005–February 2006. Morbidity and Mortality Weekly Report, 58, 329–332. Available at: (accessed September 2015).

  • Centers for Disease Control and Prevention (CDC). 2013. National HIV Behavioral Surveillance System Round 4: Model Surveillance Protocol. Available at: (accessed September 2015).

  • Cole, S.R. and M.A. Hernan. 2008. “Constructing Inverse Probability Weights for Marginal Structural Models.” American Journal of Epidemiology 168: 656–664. Doi:

  • Compton, W., J. Normand, and E. Lambert. 2009. “Sexual Acquisition and Transmission of HIV Cooperative Agreement Program (SATHCAP).” Journal of Urban Health 86(1): 1–4. Doi:

  • Constantine, M. 2010. “Disentangling Methodologies: The Ethics of Traditional Sampling Methodologies, Community-Based Participatory Research, and Respondent-Drive Sampling.” American Journal of Bioethics 10(3): 22–24. Doi:

  • Dombrowski, R., B. Khan, J. Moses, E. Channell, and E. Misshula. 2013. “Assessing Respondent Driven Sampling for Network Studies in Ethnographic Contexts.” Advances in Anthropology 3(1): 1–9. Doi:

  • Elliott, M.R. 2009. “Model Averaging Methods for Weight Trimming in Generalized Linear Regression Models.” Journal of Official Statistics 25(1): 1–21. Doi:

  • Fay, R.E. and R.A. Herriot. 1979. “Estimates of Income for Small Places: An Application of James-Stein Procedures to Census Data.” Journal of American Statistical Association 74: 269–277. Doi:

  • Frost, S.D.W., K.C. Brouwer, M.A.F. Cruz, R. Ramos, M.E. Ramos, R.M. Lozada, C. Magis-Rodriguez, and S.A. Strathdee. 2006. “Respondent-Driven Sampling of Injection Drug Users in Two U.S.-Mexico Border Cities: Recruitment Dynamics and Impact on Estimates of HIV and Syphilis Prevalence.” Journal of Urban Health 83(1): 83–97. Doi:

  • Gile, K.J. 2011. “Improved Inference for Respondent-Driven Sampling Data with Application to HIV Prevalence Estimation.” Journal of American Statistical Association 106(493): 135–146. Doi:

  • Gile, K.J. and M.S. Handcock. 2010. “Respondent-Driven Sampling: An Assessment of Current Methodology.” Sociological Methodology 40(1): 286–327. Doi:

  • Gile, K.J., L.G. Johnston, and M.J. Salganik. 2015. “Diagnostics for Respondent-Driven Sampling.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 178(1): 241–269. Doi:

  • Goel, S. and M.J. Salganik. 2010. “Assessing Respondent-Driven Sampling.” Proceedings of the National Academy of Sciences of the United States of America 107(15): 6743–6747. Doi: Available at: (accessed September 2015).

  • Groves, R.M. 1989. Survey Errors and Survey Costs. New York: Wiley.

  • Groves, R.M. and M.P. Couper. 1998. Nonresponse in Household Surveys. New York: Wiley.

  • Handcock, M.S. 2012. Estimating the Size of Hard-to-Reach Populations Using Respondent-Driven Sampling Data. Paper for the International Conference on Methods for Surveying and Enumerating Hard–to-Reach Populations, October 31–November 3, New Orleans, LA.

  • Handcock, M.S., K.J. Gile, I.E. Fellows, and W.W. Neeley. 2014. Package ‘RDS.’ Available at: (accessed September 2015).

  • Heckathorn, D.D. 1997. “Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations.” Society for the Study of Social Problems 44(2): 174–199. Doi:

  • Heckathorn, D.D. 2002. “Respondent-Driven Sampling II: Deriving Valid Population Estimates from Chain-Referral Samples of Hidden Populations.” Social Problems 49(1): 11–34. Doi:

  • Heckathorn, D.D. and J. Jeffri. 2001. “Finding the Beat: Using Respondent-Driven Sampling to Study Jazz Musicians.” Poetics 28(4): 307–329. Doi:

  • Heimer, R. 2005. “Critical Issues and Further Questions About Respondent-Driven Sampling: Comment on Ramierz-Valles et al. (2005).” AIDS and Behavior 9(4): 403–408. Doi:

  • Iguchi, M.Y., S.H. Berry, A.J. Ober, T. Fain, D.D. Heckathorn, P.M. Gorbach, R. Heimer, A. Kozlov, L.J. Ouellet, S. Shoptaw, and W. Zule. 2010. Sexual Acquisition and Transmission of HIV Cooperative Agreement Program (SATHCAP) 2006–2008 [United States]. ICPSR29181-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research. Doi:

  • Iguchi, M.Y., A.J. Ober, S.H. Berry, T. Fain, D.D. Heckathorn, P.M. Gorbach, R. Heimer, A. Kozlov, L.J. Ouellet, S. Shoptaw, and W.A. Zule. 2009. “Simultaneous Recruitment of Drug Users and Men Who Have Sex with Men in the United States and Russia Using Respondent-Driven Sampling: Sampling Methods and Implications.” Journal of Urban Health 88(1): 5–31. Doi:

  • Izrael, D., M. Battaglia, and M. Frankel. 2009. “Extreme Survey Weight Adjustment as a Component of Sample Balancing (a.k.a. Raking).” Proceedings from the 2009 SAS Global Forum. Cary, NC: SAS Institute. Available at: (accessed January 2017).

  • Johnston, L.G. and K. Sabin. 2010. “Sampling Hard-to-Reach Populations with Respondent Driven Sampling.” Methodological Innovations Online 5(2): 38–48. Doi:

  • Kish, L. 1992. “Weighting for unequal Pi.” Journal of Official Statistics 8(2): 183–200.

  • Lansky, A., A. Abdul-Quader, M. Cribbin, T. Hall, T.J. Finlayson, R.S. Garfein, L.S. Lin, and P.S. Sullivan. 2007. “Developing an HIV Behavioral Surveillance System for Injecting Drug Users: The National HIV Behavioral Surveillance System.” Public Health Reports 122: 48–55. Doi:

  • Laumann, E.O., P.V. Marsden, and D. Prensky. 1983. “The Boundary Specification Problem in Network Analysis.” In Applied Network Analysis. A Methodological Introduction, edited by R.S. Burt and M.J. Minor. 18–34. Beverly Hills, CA: Sage.

  • Lee, S. 2009. “Understanding Respondent Driven Sampling from a Total Survey Error Perspective.” Survey Practice. Available at: (accessed September 2015).

  • Lee, R., J. Ranaldi, M. Cummings, J.N. Crucetti, H. Stratton, and L.-A. McNutt. 2011. “Given the Increasing Bias in Random Digit Dial Sampling, Could Respondent-Driven Sampling be a Practical Alternative?” Annals of Epidemiology 21(4): 272–279. Doi:

  • Lee, S., Z.T. Suzer-Gurtekin, J. Wagner, and R. Valliant. 2012. Exploring Error Properties of Respondent Driven Sampling. Paper presented at the Joint Statistical Meeting, July 28–August 2, San Diego, CA.

  • Lin, L., T. Finlayson, R. Iachan, M.C.B. Mendoza, and C. Wejnert. 2013. “Sampling Designs for Populations at High Risk for HIV.” Paper presented at the Joint Statistical Meeting, August 3–August 8, Montréal, Canada.

  • Little, R.J.A., S. Lewitzky, S. Heeringa, J. Lepkowski, and R.C. Kessler. 1997. “Assessment of Weighting Methodology of the National Comorbidity Survey.” American Journal of Epidemiology 146(5): 439–449. Doi:

  • Lu, X., L. Bengtsson, T. Britton, M. Camitz, B.J. Kim, A. Thorson, and F. Liljeros. 2012. “The Sensitivity of Respondent-Driven Sampling.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 175(1): 1–26. Doi:

  • Marsden, P.V. 1990. “Network Data and Measurement.” Annual Review Sociology 16: 435–463. Doi:

  • Martin, J.L., J. Wiley, and D. Osmond. 2003. “Social Networks and Unobserved Heterogeneity in Risk for AIDS.” Population Research and Policy Review 22(1): 65–90. Doi:

  • McCarty, C., P.D. Killworth, H.R. Bernard, E.C. Johnsen, and G.A. Shelley. 2001. “Comparing Two Methods for Estimating Network Size.” Human Organization 60: 28–39. Doi:

  • McCreesh, N., S.D. Frost, J. Seeley, J. Katongole, M.N. Tarsh, R. Ndunguse, F. Jichi, N.L. Lunel, D. Maher, L.G. Johnston, P. Sonnenberg, A.J. Copas, R.J. Hayes, and R.G. White. 2012. “Evaluation of Respondent-Driven Sampling.” Epidemiology 23(1): 138–147. Doi:

  • Montealegre, J.R., J.M. Risser, B.J. Selwyn, S.A. McCurdy, and K. Sabin. 2013. “Effectiveness of Respondent Driven Sampling to Recruit Undocumented Central American Immigrant Women in Houston, Texas for an HIV Behavioral Survey.” AIDS and Behavior 17(2): 719–727. Doi:

  • Phillips, T. 2010. “Protecting the Subject: PDR and the Potential for Compromised Consent.” American Journal of Bioethics 10(3): 14–15. Doi:

  • Potter, F. 1988. “Survey of Procedures to Control Extreme Sampling Weights.” Proceedings of the Section on Survey Research Methods, American Statistical Association, 453–458. Available at: (accessed January 2017).

  • Ramirez-Valles, J. 2013. Latino MSM Community Involvement: HIV Protective Effects. ICPSR34385-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research. Doi:

  • Ramirez-Valles, J., D.D. Heckathorn, R. Vázquez, R.M. Diaz, and R.T. Campbell. 2005. “From Networks to Populations: the Development and Application of Respondent-Driven Sampling Among IDUs and Latino Gay Men.” AIDS and Behavior 9(4): 387–402. Doi:

  • Salganik, M.J. 2006. “Variance Estimation, Design Effects and Sample Size Calculations for Respondent Driven Sampling.” Journal of Urban Health 83(7): 98–112. Doi:

  • Salganik, M. 2012. “Commentary: Respondent-Driven Sampling in the Real World.” Epidemiology 23(1): 148–150. Doi:

  • Salganik, M.J. and D.D. Heckathron. 2004. “Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling.” Sociological Methodology 34: 193–239. Doi:

  • Schonlau, M. 2014. “Recruiting an Internet Panel Using Respondent Driven Sampling.” Journal of Official Statistics 30(2): 291–310. Doi:

  • Simon, C. and M. Mosavel. 2010. “Community Members as Recruiters of Human Subjects: Ethical Considerations.” American Journal of Bioethics 10(3): 3–11. Doi:

  • Sirken, M.G. 1972. “Stratified Sample Surveys with Multiplicity.” Journal of American Statistical Association 67: 224–227. Doi:

  • Sirken, M.G. 1975. “Network Surveys of Rare and Sensitive Conditions.” Advances in Health Survey Research Methods, NCHSR Research Proceedings 31. Hyattsville, MD: National Center Health Statistics.

  • Sirken, M.G. 1997. “Network Sampling.” In Encyclopedia of Biostatistics, edited by P. Armitage and T. Colton, 2977–2986. Hoboken, NJ: Wiley & Sons.

  • Valliant, R. 2013. “Comment.” Journal of Survey Statistics and Methodology 1(2): 105–111. Doi:

  • Valliant, R., J.A. Dever, and F. Kreuter. 2013. Practical Tools for Designing and Weighting Survey Samples. New York: Springer.

  • Verdery, A.M. and T.D. Mouw. 2012. Estimated Sampling Variance in Respondent Driven Sampling Data: Mathematical Derivations, Simulated Tests on Empirical Data, and Evidence from Other Forms of Chain-Referral Data Collection. Paper for the International Conference on Methods for Surveying and Enumerating Hard-to-Reach Populations, October 31–November 3, New Orleans, LA.

  • Verdery, A.M., T.D. Mouw, S. Bauldry, and P.J. Mucha. 2015. “Network Structure and Biased Variance Estimation in Respondent Driven Sampling.” PLOS ONE 10(12): e0145296. Doi:

  • Volz, E. and D.D. Heckathorn. 2008. “Probability Based Estimation Theory for Respondent Driven Sampling.” Journal of Official Statistics 24(1): 79–97.

  • Wejnert, C. and D.D. Heckathorn. 2008. “Web-Based Network Sampling: Efficiency and Efficacy of Respondent-Driven Sampling for Online Research.” Sociological Methods and Research 37: 105–134. Doi:

  • Zheng, T., M.J. Salganik, and A. Gelman. 2006. “How Many People Do You Know in Prison? Using Overdispersion in Count Data to Estimate Social Structure in Networks” Journal of American Statistical Association 101(474): 409–423. Doi:


Journal + Issues