This study attempted to integrate key assumptions in Respondent-Driven Sampling (RDS) into the Total Survey Error (TSE) perspectives and examine TSE as a new framework for a systematic assessment of RDS errors. Using two publicly available data sets on HIV-at-risk persons, nonresponse error in the RDS recruitment process and measurement error in network size reports were examined. On nonresponse, the ascertained partial nonresponse rate was high, and a substantial proportion of recruitment chains died early. Moreover, nonresponse occurred systematically: recruiters with lower income and higher health risks generated more recruits; and peers of closer relationships were more likely to accept recruitment coupons. This suggests a lack of randomness in the recruitment process, also shown through sizable intra-chain correlation. Self-reported network sizes suggested measurement error, given their wide dispersion and unreasonable reports. This measurement error has further implications for the current RDS estimators, which use network sizes as an adjustment factor on the assumption of a positive relationship between network sizes and selection probabilities in recruitment. The adjustment resulted in nontrivial unequal weighting effects and changed estimates in directions that were difficult to explain and, at times, illogical. Moreover, recruiters’ network size played no role in actual recruitment. TSE may serve as a tool for evaluating errors in RDS, which further informs study design decisions and inference approaches.
If the inline PDF is not rendering correctly, you can download the PDF file here.
Baker R. J.M. Brick N.A. Bates M. Battaglia M.P. Couper J.A. Dever K.J. Gile and R. Tourangeau. 2013. “Summary Report of the AAPOR Task Force on Non-Probability Sampling.” Journal of Survey Statistics and Methodology 1(2): 90–143. Doi: https://doi.org/10.1093/jssam/smt008.
Beaumont J.-F. 2008. “A New Approach to Weighting and Inference in Sample Surveys.” Biometrika 95(3): 539–553. Doi: https://doi.org/10.1093/biomet/asn028.
Burt R.D. H. Hagan K. Sabin and H. Thiede. 2010. “Evaluating Respondent-Driven Sampling in a Major Metropolitan Area: Comparing Injection Drug Users in the 2005 Seattle Area National HIV Behavioral Surveillance System Survey with Participants in the RACEN and Kiwi Studies.” Annals of Epidemiology 20(2): 159–167. Doi: https://doi.org/10.1016/j.annepidem.2009.10.002.
Centers for Disease Control and Prevention (CDC). 2009. HIV-Associated Behaviors Among Injecting-Drug Users––23 Cities United States May 2005–February 2006. Morbidity and Mortality Weekly Report 58 329–332. Available at: http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5813a1.htm (accessed September 2015).
Centers for Disease Control and Prevention (CDC). 2013. National HIV Behavioral Surveillance System Round 4: Model Surveillance Protocol. Available at: http://www.cdc.gov/hiv/pdf/NHBS_Round4ModelSurveillanceProtocol.pdf (accessed September 2015).
Cole S.R. and M.A. Hernan. 2008. “Constructing Inverse Probability Weights for Marginal Structural Models.” American Journal of Epidemiology 168: 656–664. Doi: https://doi.org/10.1093/aje/kwn164.
Compton W. J. Normand and E. Lambert. 2009. “Sexual Acquisition and Transmission of HIV Cooperative Agreement Program (SATHCAP).” Journal of Urban Health 86(1): 1–4. Doi: https://doi.org/10.1007/s11524-009-9373-4.
Constantine M. 2010. “Disentangling Methodologies: The Ethics of Traditional Sampling Methodologies Community-Based Participatory Research and Respondent-Drive Sampling.” American Journal of Bioethics 10(3): 22–24. Doi: https://doi.org/10.1080/15265160903585628.
Dombrowski R. B. Khan J. Moses E. Channell and E. Misshula. 2013. “Assessing Respondent Driven Sampling for Network Studies in Ethnographic Contexts.” Advances in Anthropology 3(1): 1–9. Doi: https://doi.org/10.4236/aa.2013.31001.
Elliott M.R. 2009. “Model Averaging Methods for Weight Trimming in Generalized Linear Regression Models.” Journal of Official Statistics 25(1): 1–21. Doi: https://doi.org/10.1.1.552.9050.
Fay R.E. and R.A. Herriot. 1979. “Estimates of Income for Small Places: An Application of James-Stein Procedures to Census Data.” Journal of American Statistical Association 74: 269–277. Doi: https://doi.org/10.2307/2286322.
Frost S.D.W. K.C. Brouwer M.A.F. Cruz R. Ramos M.E. Ramos R.M. Lozada C. Magis-Rodriguez and S.A. Strathdee. 2006. “Respondent-Driven Sampling of Injection Drug Users in Two U.S.-Mexico Border Cities: Recruitment Dynamics and Impact on Estimates of HIV and Syphilis Prevalence.” Journal of Urban Health 83(1): 83–97. Doi: https://doi.org/10.1007/s11524-006-9104-z.
Gile K.J. 2011. “Improved Inference for Respondent-Driven Sampling Data with Application to HIV Prevalence Estimation.” Journal of American Statistical Association 106(493): 135–146. Doi: https://doi.org/10.1198/jasa.2011.ap09475.
Gile K.J. and M.S. Handcock. 2010. “Respondent-Driven Sampling: An Assessment of Current Methodology.” Sociological Methodology 40(1): 286–327. Doi: https://doi.org/10.1111/j.1467-9531.2010.01223.x.
Gile K.J. L.G. Johnston and M.J. Salganik. 2015. “Diagnostics for Respondent-Driven Sampling.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 178(1): 241–269. Doi: https://doi.org/10.1111/rssa.12059.
Goel S. and M.J. Salganik. 2010. “Assessing Respondent-Driven Sampling.” Proceedings of the National Academy of Sciences of the United States of America 107(15): 6743–6747. Doi: https://doi.org/10.1073/pnas.1000261107. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2872407/ (accessed September 2015).
Groves R.M. 1989. Survey Errors and Survey Costs. New York: Wiley.
Groves R.M. and M.P. Couper. 1998. Nonresponse in Household Surveys. New York: Wiley.
Handcock M.S. 2012. Estimating the Size of Hard-to-Reach Populations Using Respondent-Driven Sampling Data. Paper for the International Conference on Methods for Surveying and Enumerating Hard–to-Reach Populations October 31–November 3 New Orleans LA.
Handcock M.S. K.J. Gile I.E. Fellows and W.W. Neeley. 2014. Package ‘RDS.’ Available at: http://cran.r-project.org/web/packages/RDS/RDS.pdf (accessed September 2015).
Heckathorn D.D. 1997. “Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations.” Society for the Study of Social Problems 44(2): 174–199. Doi: https://doi.org/10.2307/3096941.
Heckathorn D.D. 2002. “Respondent-Driven Sampling II: Deriving Valid Population Estimates from Chain-Referral Samples of Hidden Populations.” Social Problems 49(1): 11–34. Doi: https://doi.org/10.1525/sp.2002.49.1.11.
Heckathorn D.D. and J. Jeffri. 2001. “Finding the Beat: Using Respondent-Driven Sampling to Study Jazz Musicians.” Poetics 28(4): 307–329. Doi: https://doi.org/10.1016/S0304-422X(01)80006-1.
Heimer R. 2005. “Critical Issues and Further Questions About Respondent-Driven Sampling: Comment on Ramierz-Valles et al. (2005).” AIDS and Behavior 9(4): 403–408. Doi: https://doi.org/10.1007/s10461-005-9030-1.
Iguchi M.Y. S.H. Berry A.J. Ober T. Fain D.D. Heckathorn P.M. Gorbach R. Heimer A. Kozlov L.J. Ouellet S. Shoptaw and W. Zule. 2010. Sexual Acquisition and Transmission of HIV Cooperative Agreement Program (SATHCAP) 2006–2008 [United States]. ICPSR29181-v1. Ann Arbor MI: Inter-university Consortium for Political and Social Research. Doi: https://doi.org/10.3886/ICPSR29181.
Iguchi M.Y. A.J. Ober S.H. Berry T. Fain D.D. Heckathorn P.M. Gorbach R. Heimer A. Kozlov L.J. Ouellet S. Shoptaw and W.A. Zule. 2009. “Simultaneous Recruitment of Drug Users and Men Who Have Sex with Men in the United States and Russia Using Respondent-Driven Sampling: Sampling Methods and Implications.” Journal of Urban Health 88(1): 5–31. Doi: https://doi.org/10.1007/s11524-009-9365-4.
Izrael D. M. Battaglia and M. Frankel. 2009. “Extreme Survey Weight Adjustment as a Component of Sample Balancing (a.k.a. Raking).” Proceedings from the 2009 SAS Global Forum. Cary NC: SAS Institute. Available at: http://abtassociates.com/AbtAssociates/files/c1/c1bc376c-1931-4721-b71c-cb823a0fe809.pdf (accessed January 2017).
Johnston L.G. and K. Sabin. 2010. “Sampling Hard-to-Reach Populations with Respondent Driven Sampling.” Methodological Innovations Online 5(2): 38–48. Doi: https://doi.org/10.4256/mio.2010.0017.
Kish L. 1992. “Weighting for unequal Pi.” Journal of Official Statistics 8(2): 183–200.
Lansky A. A. Abdul-Quader M. Cribbin T. Hall T.J. Finlayson R.S. Garfein L.S. Lin and P.S. Sullivan. 2007. “Developing an HIV Behavioral Surveillance System for Injecting Drug Users: The National HIV Behavioral Surveillance System.” Public Health Reports 122: 48–55. Doi: https://doi.org/10.1177/00333549071220S108.
Laumann E.O. P.V. Marsden and D. Prensky. 1983. “The Boundary Specification Problem in Network Analysis.” In Applied Network Analysis. A Methodological Introduction edited by R.S. Burt and M.J. Minor. 18–34. Beverly Hills CA: Sage.
Lee S. 2009. “Understanding Respondent Driven Sampling from a Total Survey Error Perspective.” Survey Practice. Available at: http://www.surveypractice.org/index.php/SurveyPractice/article/view/187/html (accessed September 2015).
Lee R. J. Ranaldi M. Cummings J.N. Crucetti H. Stratton and L.-A. McNutt. 2011. “Given the Increasing Bias in Random Digit Dial Sampling Could Respondent-Driven Sampling be a Practical Alternative?” Annals of Epidemiology 21(4): 272–279. Doi: https://doi.org/10.1016/j.annepidem.2010.11.018.
Lee S. Z.T. Suzer-Gurtekin J. Wagner and R. Valliant. 2012. Exploring Error Properties of Respondent Driven Sampling. Paper presented at the Joint Statistical Meeting July 28–August 2 San Diego CA.
Lin L. T. Finlayson R. Iachan M.C.B. Mendoza and C. Wejnert. 2013. “Sampling Designs for Populations at High Risk for HIV.” Paper presented at the Joint Statistical Meeting August 3–August 8 Montréal Canada.
Little R.J.A. S. Lewitzky S. Heeringa J. Lepkowski and R.C. Kessler. 1997. “Assessment of Weighting Methodology of the National Comorbidity Survey.” American Journal of Epidemiology 146(5): 439–449. Doi: https://doi.org/10.1093/oxfordjournals.aje.a009297.
Lu X. L. Bengtsson T. Britton M. Camitz B.J. Kim A. Thorson and F. Liljeros. 2012. “The Sensitivity of Respondent-Driven Sampling.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 175(1): 1–26. Doi: https://doi.org/10.1111/j.1467-985X.2011.00711.x.
Marsden P.V. 1990. “Network Data and Measurement.” Annual Review Sociology 16: 435–463. Doi: https://doi.org/10.1146/annurev.so.16.080190.002251.
Martin J.L. J. Wiley and D. Osmond. 2003. “Social Networks and Unobserved Heterogeneity in Risk for AIDS.” Population Research and Policy Review 22(1): 65–90. Doi: https://doi.org/10.1023/A:1023509211339.
McCarty C. P.D. Killworth H.R. Bernard E.C. Johnsen and G.A. Shelley. 2001. “Comparing Two Methods for Estimating Network Size.” Human Organization 60: 28–39. Doi: https://doi.org/10.17730/humo.60.1.efx5t9gjtgmga73y.
McCreesh N. S.D. Frost J. Seeley J. Katongole M.N. Tarsh R. Ndunguse F. Jichi N.L. Lunel D. Maher L.G. Johnston P. Sonnenberg A.J. Copas R.J. Hayes and R.G. White. 2012. “Evaluation of Respondent-Driven Sampling.” Epidemiology 23(1): 138–147. Doi: https://doi.org/10.1097/EDE.0b013e31823ac17c.
Montealegre J.R. J.M. Risser B.J. Selwyn S.A. McCurdy and K. Sabin. 2013. “Effectiveness of Respondent Driven Sampling to Recruit Undocumented Central American Immigrant Women in Houston Texas for an HIV Behavioral Survey.” AIDS and Behavior 17(2): 719–727. Doi: https://doi.org/10.1007/s10461-012-0306-y.
Phillips T. 2010. “Protecting the Subject: PDR and the Potential for Compromised Consent.” American Journal of Bioethics 10(3): 14–15. Doi: https://doi.org/10.1080/15265160903585602.
Potter F. 1988. “Survey of Procedures to Control Extreme Sampling Weights.” Proceedings of the Section on Survey Research Methods American Statistical Association 453–458. Available at: http://www.websm.org/uploadi/editor/1368363852Potter_1988_Survey_of_procedures_to_control_extreme_sampling_weights.pdf (accessed January 2017).
Ramirez-Valles J. 2013. Latino MSM Community Involvement: HIV Protective Effects. ICPSR34385-v1. Ann Arbor MI: Inter-university Consortium for Political and Social Research. Doi: https://doi.org/10.3886/ICPSR34385.v1.
Ramirez-Valles J. D.D. Heckathorn R. Vázquez R.M. Diaz and R.T. Campbell. 2005. “From Networks to Populations: the Development and Application of Respondent-Driven Sampling Among IDUs and Latino Gay Men.” AIDS and Behavior 9(4): 387–402. Doi: https://doi.org/10.1007/s10461-005-9012-3.
Salganik M.J. 2006. “Variance Estimation Design Effects and Sample Size Calculations for Respondent Driven Sampling.” Journal of Urban Health 83(7): 98–112. Doi: https://doi.org/10.1007/s11524-006-9106-x.
Salganik M. 2012. “Commentary: Respondent-Driven Sampling in the Real World.” Epidemiology 23(1): 148–150. Doi: https://doi.org/10.1097/EDE.0b013e31823b6979.
Salganik M.J. and D.D. Heckathron. 2004. “Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling.” Sociological Methodology 34: 193–239. Doi: https://doi.org/10.1111/j.0081-1750.2004.00152.x.
Schonlau M. 2014. “Recruiting an Internet Panel Using Respondent Driven Sampling.” Journal of Official Statistics 30(2): 291–310. Doi: https://doi.org/10.2478/jos-2014-0018.
Simon C. and M. Mosavel. 2010. “Community Members as Recruiters of Human Subjects: Ethical Considerations.” American Journal of Bioethics 10(3): 3–11. Doi: https://doi.org/10.1080/15265160903585578.
Sirken M.G. 1972. “Stratified Sample Surveys with Multiplicity.” Journal of American Statistical Association 67: 224–227. Doi: https://doi.org/10.1080/01621459.1972.10481236.
Sirken M.G. 1975. “Network Surveys of Rare and Sensitive Conditions.” Advances in Health Survey Research Methods NCHSR Research Proceedings 31. Hyattsville MD: National Center Health Statistics.
Sirken M.G. 1997. “Network Sampling.” In Encyclopedia of Biostatistics edited by P. Armitage and T. Colton 2977–2986. Hoboken NJ: Wiley & Sons.
Valliant R. 2013. “Comment.” Journal of Survey Statistics and Methodology 1(2): 105–111. Doi: https://doi.org/10.1093/jssam/smt010.
Valliant R. J.A. Dever and F. Kreuter. 2013. Practical Tools for Designing and Weighting Survey Samples. New York: Springer.
Verdery A.M. and T.D. Mouw. 2012. Estimated Sampling Variance in Respondent Driven Sampling Data: Mathematical Derivations Simulated Tests on Empirical Data and Evidence from Other Forms of Chain-Referral Data Collection. Paper for the International Conference on Methods for Surveying and Enumerating Hard-to-Reach Populations October 31–November 3 New Orleans LA.
Verdery A.M. T.D. Mouw S. Bauldry and P.J. Mucha. 2015. “Network Structure and Biased Variance Estimation in Respondent Driven Sampling.” PLOS ONE 10(12): e0145296. Doi: http://dx.doi.org/10.1371/journal.pone.0145296.
Volz E. and D.D. Heckathorn. 2008. “Probability Based Estimation Theory for Respondent Driven Sampling.” Journal of Official Statistics 24(1): 79–97.
Wejnert C. and D.D. Heckathorn. 2008. “Web-Based Network Sampling: Efficiency and Efficacy of Respondent-Driven Sampling for Online Research.” Sociological Methods and Research 37: 105–134. Doi: https://doi.org/10.1177/0049124108318333.
Zheng T. M.J. Salganik and A. Gelman. 2006. “How Many People Do You Know in Prison? Using Overdispersion in Count Data to Estimate Social Structure in Networks” Journal of American Statistical Association 101(474): 409–423. Doi: https://doi.org/10.1198/01621450500000116.