In this article, we review current state-of-the art software enabling statisticians to apply design-based, model-based, and so-called “hybrid” approaches to the analysis of complex sample survey data. We present brief overviews of the similarities and differences between these alternative approaches, and then focus on software tools that are presently available for implementing each approach. We conclude with a summary of directions for future software development in this area.
Archer, K.J., S. Lemeshow, and D.W. Hosmer. 2007. “Goodness-of-fit Tests for Logistic Regression Models When Data are Collected using a Complex Sampling Design.” Computational Statistics and Data Analysis 51: 4450–4464. Doi: https://doi.org/10.1016/j.csda.2006.07.006.
Asparouhov, T. and B. Muthén. 2007. “Testing for Informative Weights and Weights Trimming in Multivariate Modelling with Survey Data.” In Proceedings of the Survey Research Methods Section of the American Statistical Association, 2007, Salt Lake City, Utah, 3394–3399. Available at: https://www.statmodel.com/download/JSM2007000745.pdf (accessed April 14, 2017).
Baker, R., J.M. Brick, N.A. Bates, M. Battaglia, M.P. Couper, J.A. Dever, and R. Tourangeau. 2013. “Summary Report of the AAPOR Task Force on Non-probability Sampling.” Journal of Survey Statistics and Methodology 1: 90–143. Doi: https://doi.org/10.1093/jssam/smt008.
Barnighausen, T., J. Bor, S. Wandira-Kazibwe, and D. Canning. 2011. “Correcting HIV Prevalence Estimates for Survey Nonparticipation using Heckman-type Selection Models.” Epidemiology 22: 27–35. Doi: 10.1097/EDE.0b013e3181ffa201.
Bean, J.A. 1975. “Distribution and Properties of Variance Estimators for Complex Multistage Probability Samples: An Empirical Distribution.” In Vital and Health Statistics: Series 2, Data Evaluation and Methods Research 65: i–iv.
Beaumont, J.F. 2005. “On the Use of Data Collection Process Information for the Treatment of Unit Nonresponse Through Weight Adjustment.” Survey Methodology 31: 227–231.
Bethlehem, J.G. 2002. “Weighting Nonresponse Adjustments Based on Auxiliary Information.” In Survey Nonresponse, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 275–288. New York: Wiley.
Binder, D.A. 1981. “On the Variances of Asymptotically Normal Estimators for Complex Surveys.” Survey Methodology 7: 157–170.
Binder, D.A. 1983. “On the Variances of Asymptotically Normal Estimators from Complex Surveys.” International Statistical Review 51: 279–292. Doi: 10.2307/1402588.
Binder, D.A. and G.R. Roberts. 2003. “Design-based and Model-based Methods for Estimating Model Parameters.” In Analysis of Survey Data, edited by R.L. Chambers and C.J. Skinner, 29–48. Chichester, West Sussex: Wiley.
Bollen, K.A., P.P. Biemer, A.F. Karr, S. Tueller, and M.E. Berzofsky. 2016. “Are Survey Weights Needed? A Review of Diagnostic Tests in Regression Analysis.” Annual Review of Statistics and Its Application 3: 375–392. Doi: https://doi.org/10.1146/annurev-statistics-011516-012958.
Breckling, J.U., R.L. Chambers, A.H. Dorfman, S.M. Tam, and A.H. Welsh. 1994. “Maximum Likelihood Inference from Sample Survey Data.” International Statistical Review 62: 349–363. Doi: 10.2307/1403766.
Canty, A.J. and A.C. Davison. 1999. “Resampling-based Variance Estimation for Labour Force Surveys.” The Statistician 48: 379–391. Doi: 10.1111/1467-9884.00196.
Carle, A.C. 2009. “Fitting Multilevel Models in Complex Survey Data with Design Weights: Recommendations.” BMC Medical Research Methodology 9(49). Doi: https://doi.org/10.1186/1471-2288-9-49.
Cassel, C., C.-E. Särndal, and J. Wretman. 1983. “Some Uses of Statistical Models in Connection with the Nonresponse Problem.” In Incomplete Data in Sample Surveys, edited by W.G. Madow and I. Olkin, 143–160. New York: Academic Press.
Chambers, R.L., A.H. Dorfman, and S. Wang. 1998. “Limited Information Likelihood Analysis of Survey Data.” Journal of the Royal Statistical Society (Series B) 60: 397–411. Doi: 10.1111/1467-9868.00132.
Chambers, R.L. and C.J. Skinner (Editors). 2003. Analysis of Survey Data. New York: John Wiley and Sons.
Chambless, L.E. and K.E. Boyle. 1985. “Maximum Likelihood Methods for Complex Sample Data: Logistic Regression and Discrete Proportional Hazards Models.” Communications in Statistics-Theory and Methods 14: 1377–1392. Doi: https://doi.org/10.1080/03610928508828982.
Claeskens, G. 2013. “Lack of Fit, Graphics, and Multilevel Model Diagnostics.” In The SAGE Handbook of Multilevel Modeling, edited by M.A. Scott, J.S. Simonoff, and B.D. Marx, 425–444. Los Angeles: SAGE Publications.
Dean, N. and M. Pagano. 2015. “Evaluating Confidence Interval Methods for Binomial Proportions in Clustered Surveys.” Journal of Survey Statistics and Methodology 3: 484–503. Doi: https://doi.org/10.1093/jssam/smv024.
DuMouchel, W.H. and G.J. Duncan. 1983. “Using Sample Survey Weights in Multiple Regression Analyses of Stratified Samples.” Journal of the American Statistical Association 78: 535–543. Doi: https://doi.org/10.1080/01621459.1983.10478006.
Ekholm, A. and S. Laaksonen. 1991. “Weighting Via Response Modeling in the Finnish Household Budget Survey.” Journal of Official Statistics 7: 325–337.
Elliott, M.R. and R.J. Little. 2000. “Model-based Alternatives to Trimming Survey Weights.” Journal of Official Statistics 16: 191–210.
Elliott, M.R. and R. Valliant. 2017. “Inference for Nonprobability Samples.” Statistical Science 32: 249–264. Doi: 10.1214/16-STS598.
Eltinge, J.L. and I.S. Yansaneh. 1997. “Diagnostics for Formation of Nonresponse Adjustment Cells, With an Application to Income Nonresponse in the U.S. Consumer Expenditure Survey.” Survey Methodology 23: 33–40.
Feder, M. 2011. “Fitting Regression Models to Complex Survey Data – Gelman’s Estimator Revisited.” In Proceedings of the 58th World Statistics Congress of the International Statistical Institute, Dublin, Ireland, August 2011. Available at: http://2011.isiproceedings.org/papers/950551.pdf (accessed January 30, 2018).
Flores-Cervantes, I. and J.M. Brick. 2016. “Nonresponse Adjustments with Misspecified Models in Stratified Designs.” Survey Methodology 42: 161–177.
Frankel, M.R. 1971. Inference from Survey Samples: an Empirical Investigation. Institute for Social Research, University of Michigan, Ann Arbor, MI, USA.
Gagne, C., G. Roberts, and L.-A. Keown. 2014. “Weighted Estimation and Bootstrap Variance Estimation for Analyzing Survey Data: How to Implement in Selected Software.” Statistics Canada: The Research Data Centres Information and Technical Bulletin, August 7, 2014. Available at: http://www.statcan.gc.ca/pub/12-002-x/2014001/article/11901-eng.htm (accessed January 30, 2018).
Gelman, A. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science 22: 153–164. Doi: 10.1214/088342306000000691.
Gelman, A., J.B. Carlin, H.S. Stern, and D.B. Rubin. 2003. Bayesian Data Analysis (2nd Edition). Boca Raton, FL: Chapman & Hall/CRC.
Hansen, M.H., W.G. Madow, and B.J. Tepping. 1983. “An Evaluation of Model-Dependent and Probability-Sampling Inferences in Sample Surveys.” Journal of the American Statistical Association 78: 776–793. Doi: https://doi.org/10.1080/01621459.1983.10477018.
Hosmer, D.W., S. Lemeshow, and X. Sturdivant. 2013. Applied Logistic Regression, Third Edition. New York, NY: Wiley.
Judkins, D.R. 1990. “Fay’s Method for Variance Estimation.” Journal of Official Statistics 6: 223–239.
Kalton, G. and I. Flores-Cervantes. 2003. “Weighting Methods.” Journal of Official Statistics 19: 81–97.
Kaplan, D. and A.J. Ferguson. 1999. “On the Utilization of Sample Weights in Latent Variable Models.” Structural Equation Modeling: A Multidisciplinary Journal 6: 305–321. Doi: https://doi.org/10.1080/10705519909540138.
Korn, E.L. and B.I. Graubard. 1999. Analysis of Health Surveys. New York, NY: Wiley.
Kovačević, M.S. and S.N. Rai. 2003. “A Pseudo Maximum Likelihood Approach to Multilevel Modelling of Survey Data.” Communications in Statistics-Theory and Methods 32: 103–121. Doi: https://doi.org/10.1081/STA-120017802.
Kovar, J.G., J.N.K. Rao, and C.F.J. Wu. 1988. “Bootstrap and Other Methods to Measure Errors in Survey Estimates.” Canadian Journal of Statistics 16: 25–45. Doi: 10.2307/3315214.
Kreuter, F., K. Olson, J. Wagner, T. Yan, T.M. Ezzati-Rice, C. Casas-Cordero, M. Lemay, A. Peytchev, R.M. Groves, and T.E. Raghunathan. 2010. “Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Non-Response: Examples from Multiple Surveys.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 173: 389–407. Doi: 10.1111/j.1467-985X.2009.00621.x.
Krewski, D. and J.N.K. Rao. 1981. “Inference from Stratified Samples: Properties of the Linearization, Jackknife and Balanced Repeated Replication Methods.” The Annals of Statistics 9: 1010–1019. Doi: http://www.jstor.org/stable/2240615.
Lavallée, P. and J.F. Beaumont. 2016. “Weighting: Principles and Practicalities.” In The SAGE Handbook of Survey Methodology, edited by C. Wolf, D. Joye, T.W. Smith, and Y. Fu, 460–476. London: Sage.
Lemeshow, S. and P. Levy. 1978. “Estimating the Variance of Ratio Estimates in Complex Sample Surveys with Two Primary Units per Stratum––A Comparison of Balanced Replication and Jackknife Techniques.” Journal of Statistical Computation and Simulation 8: 191–205. Doi: https://doi.org/10.1080/00949657908810266.
Lessler, J.T. and W.D. Kalsbeek. 1992. Nonsampling Error in Surveys. Wiley.
Li, J. and R. Valliant. 2009. “Survey Weighted Hat Matrix and Leverages.” Survey Methodology 35: 15–24.
Li, J. and R. Valliant. 2011a. “Linear Regression Influence Diagnostics for Unclustered Survey Data.” Journal of Official Statistics 27: 99–119.
Li, J. and R. Valliant. 2011b. “Detecting Groups of Influential Observations in Linear Regression using Survey Data: Adapting the Forward Search Method.” Pakistan Journal of Statistics 27: 507–528.
Little, R.J.A. 2003. “The Bayesian Approach to Sample Survey Inference.” In Analysis of Survey Data, edited by R.L. Chambers and C.J. Skinner, 49–57. Chichester, West Sussex: Wiley.
Little, R.J.A. 2004. “To Model or Not to Model? Competing Modes of Inference for Finite Population Sampling.” Journal of the American Statistical Association 99: 546–556. Doi: https://doi.org/10.1198/016214504000000467.
Lumley, T. 2010. Complex Surveys: A Guide to Analysis Using R. New York, NY: Wiley.
Lumley, T. and A. Scott. 2013. “Partial Likelihood Ratio Tests for the Cox Model under Complex Sampling.” Statistics in Medicine 32: 110–123. Doi: https://doi.org/10.1002/sim.5492.
Lumley, T. and A. Scott. 2014. “Tests for Regression Models Fitted to Survey Data.” Australian & New Zealand Journal of Statistics 56: 1–14. Doi: https://doi.org/10.1111/anzs.12065.
Lumley, T. and A. Scott. 2015. “AIC and BIC for Modeling with Complex Survey Data.” Journal of Survey Statistics and Methodology 3: 1–18. Doi: https://doi.org/10.1093/jssam/smu021.
Lumley, T. and A. Scott. 2017. “Fitting Regression Models to Survey Data.” Statistical Science 32: 265–278. Doi: https://10.1214/16-STS605.
Lundström, S. and C.E. Särndal. 1999. “Calibration as a Standard Method for Treatment of Nonresponse.” Journal of Official Statistics 15: 305–327.
Lunn, D., C. Jackson, N. Best, A. Thomas, and D. Spiegelhalter. 2012. The BUGS book: A Practical Introduction to Bayesian Analysis. CRC press.
Morel, G. 1989. “Logistic Regression under Complex Survey Designs.” Survey Methodology 15: 203–223.
Muthén, B.O. and A. Satorra. 1995. “Complex Sample Data in Structural Equation Modeling.” Sociological Methodology 25: 267–316. Doi: https://doi.org/10.2307/271070.
Nordberg, L. 1989. “Generalized Linear Modeling of Sample Survey Data.” Journal of Official Statistics 5: 223–239.
Oberski, D.L. 2014. “lavaan.survey: An R package for Complex Survey Analysis of Structural Equation Models.” Journal of Statistical Software 57: 1–27. Doi: https://doi.org/10.18637/jss.v057.i01.
Oh, H.L. and F.J. Scheuren. 1983. “Weighting Adjustment for Unit Nonresponse.” In Incomplete Data in Sample Surveys, edited by W.G. Madow, I. Olkin, and D.B. Rubin, 143–184. New York: Academic Press.
Orchard, T. and M.A. Woodbury. 1972. “A Missing Information Principle: Theory and Applications.” Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Theory of Statistics, 697–715. University of California Press: Berkeley, CA. Available at: https://projecteuclid.org/download/pdf_1/euclid.bsmsp/1200514117 (accessed January 30, 2018).
Owen, A.B. 2001. Empirical Likelihood. New York: Chapman & Hall.
Pacifico, D. 2014. “sreweight: A Stata Command to Reweight Survey Data to External Totals.” The Stata Journal 14: 4–21.
Pfeffermann, D. and A. Sikov. 2011. “Imputation and Estimation under Nonignorable Nonresponse in Household Surveys with Missing Covariate Information.” Journal of Official Statistics 27: 181–209.
Pfeffermann, D., C.J. Skinner, D.J. Holmes, H. Goldstein, and J. Rasbash. 1998. “Weighting for Unequal Selection Probabilities in Multilevel Models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60: 23–40. Doi: https://doi.org/10.1111/1467-9868.00106.
Pfeffermann, D. and M. Sverchkov. 2009. “Inference Under Informative Sampling.” In Handbook of Statistics – Sample Surveys: Inference and Analysis (Volume 29, Part B), edited by V.N. Gudivada, V.V. Raghavan, V. Govindaraju, and C.R. Rao, 455–487.
Rao, J.N.K. and A.J. Scott. 1981. “The Analysis of Categorical Data from Complex Sample Surveys: Chi-squared Tests for Goodness of Fit and Independence in Two-way Tables.” Journal of the American Statistical Association 76: 221–230. Doi: https://doi.org/10.2307/2287815.
Rao, J.N.K. and A.J. Scott. 1984. “On Chi-squared Tests for Multiway Contingency Tables with Cell Proportions Estimated from Survey Data.” The Annals of Statistics 12: 46–60. Doi: http://dx.doi.org/10.1214/aos/1176346391.
Rao, J.N.K. and C.F.J. Wu. 1985. “Inference from Stratified Samples: Second-order Analysis of Three Methods for Nonlinear Statistics.” Journal of the American Statistical Association 80: 620–630. Doi: https://doi.org/10.2307/2288478.
Rao, J.N.K. and C.F.J. Wu. 1987. “Methods for Standard Errors and Confidence Intervals from Sample Survey Data: Some Recent Work.” Bulletin of the International Statistical Institute 3: 5–21.
Rao, J.N.K. and C.F.J. Wu. 1988. “Resampling Inference with Complex Survey Data.” Journal of the American Statistical Association 83: 231–241. Doi: https://doi.org/10.2307/2288945.
Reiter, J.P., T.E. Raghunathan, and S.K. Kinney. 2006. “The Importance of Modeling the Sampling Design in Multiple Imputation for Missing Data.” Survey Methodology 32: 143–149.
Roberts, G., J.N.K. Rao, and S. Kumar. 1987. “Logistic Regression Analysis of Sample Survey Data.” Biometrika 74: 1–12. Doi: https://doi.org/10.2307/2336016.
Rubin, D.B. 1985. “The Use of Propensity Scores in Applied Bayesian Inference.” In Bayesian Statistics 2, edited by J.M. Bernardo, M.H. Degroot, D.V. Lindley, and A.F.M. Smith, 463–472. Elsevier Science Publishers B.V.
Ryan, B.L., J. Koval, B. Corbett, A. Thind, M.K. Campbell, and M. Stewart. 2015. “Assessing the Impact of Potentially Influential Observations in Weighted Logistic Regression.” The Research and Data Centres Information and Technical Bulletin (Statistics Canada) 7. Available at: http://www.statcan.gc.ca/pub/12-002-x/2015001/article/14147-eng.htm (accessed January 30, 2018).
Särndal, C.E., B. Swensson, and J. Wretman. 1992. Model Assisted Survey Sampling. New York: Springer-Verlag Inc.
Särndal, C.E. and B. Swensson. 1987. “A General View of Estimation for Two Phases of Selection with Applications to Two-Phase Sampling and Nonresponse.” International Statistical Review 55: 279–294. Doi: https://doi.org/10.2307/1403406.
Shah, B.V., M.M. Holt, and R.E. Folsom. 1977. “Inference about Regression Models from Sample Survey Data.” Bulletin of the International Statistical Institute 47: 43–57.
Shao, J. 2003. “Impact of the Bootstrap on Sample Surveys.” Statistical Science 18: 191–198.
Shao, J. and R.R. Sitter. 1996. “Bootstrap for Imputed Survey Data.” Journal of the American Statistical Association 91: 1278–1288. Doi: https://doi.org/10.2307/2291746.
Shao, J. and D. Tu. 1995. The Jackknife and Bootstrap. New York: Springer.
Valliant, R. 2004. “The Effect of Multiple Weighting Steps on Variance Estimation.” Journal of Official Statistics 20(1): 1–18.
Valliant, R., J.A. Dever, and F. Kreuter. 2013. Practical Tools for Designing and Weighting Survey Samples. New York: Springer.
Valliant, R., A.H. Dorfman, and R.M. Royall. 2000. Finite Population Sampling and Inference: a Prediction Approach. New York: Wiley.
Valliant, R. and K.F. Rust. 2010. “Degrees of Freedom Approximations and Rules-of-Thumb.” Journal of Official Statistics 26: 585–602.
West, B.T. 2009. “A Simulation Study of Alternative Weighting Class Adjustments for Nonresponse when Estimating a Population Mean from Complex Sample Survey Data.” In Proceedings of the section on Survey Research Methods: Joint Statistical Meetings, 4920–4933. Available at: http://ww2.amstat.org/sections/srms/Proceedings/y2009/Files/305394.pdf (accessed January 30, 2018).
West, B.T., L. Beer, W. Gremel, J. Weiser, C. Johnson, S. Garg, and J. Skarbinski. 2015. “Weighted Multilevel Models: A Case Study.” American Journal of Public Health 105: 2214–2215. Doi: https://dx.doi.org/10.2105%2FAJPH.2015.302842.
West, B.T., P.A. Berglund, and S.G. Heeringa. 2008. “A Closer Examination of Subpopulation Analysis of Complex-Sample Survey Data.” The Stata Journal 8: 520–531.
West, B.T. and S.E. McCabe. 2017. “Alternative Approaches to Assessing Nonresponse Bias in Longitudinal Survey Estimates: An Application to Substance Use Outcomes among Young Adults in the U.S.” American Journal of Epidemiology 185: 591–600. Doi: https://doi.org/10.1093/aje/kww115.
Wun, L.M., T.M. Ezzati-Rice, N. Diaz-Tena, and J. Greenblatt. 2007. “On Modeling Response Propensity for Dwelling Unit (DU) Level Non-response Adjustment in the Medical Expenditure Panel Survey (MEPS).” Statistics in Medicine 26: 1875–1884. Doi: https://doi.org/10.1002/sim.2809.
Zangeneh, S.Z. and R.J. Little. 2015. “Bayesian Inference for the Finite Population Total from a Heteroscedastic Probability Proportional to Size Sample.” Journal of Survey Statistics and Methodology 3: 162–192. Doi: https://doi.org/10.1093/jssam/smv002.
Zhang, D. and X. Lin. 2008. “Variance Component Testing in Generalized Linear Mixed Models for Longitudinal/Clustered Data and Other Related Topics.” In Random Effect and Latent Variable Model Selection, edited by D.B. Dunson. Springer Lecture Notes in Statistics, 192.
Zheng, H. and R.J. Little. 2003. “Penalized Spline Model-Based Estimation of the Finite Populations Total from Probability-Proportional-to-Size Samples.” Journal of Official Statistics 19: 99–117.
Zhou, H., M.R. Elliott, and T.E. Raghunathan. 2016a. “Synthetic Multiple-Imputation Procedure for Multistage Complex Samples.” Journal of Official Statistics 32: 231–256. Doi: https://doi.org/10.1515/JOS-2016-0011.
Zhou, H., M.R. Elliott, and T.E. Raghunathan. 2016b. “Multiple Imputation in Two-Stage Cluster Samples Using the Weighted Finite Population Bayesian Boostrap.” Journal of Survey Statistics and Methodology 4: 139–170. Doi: https://doi.org/10.1093/jssam/smv031.
Zhou, H., M.R. Elliott, and T.E. Raghunathan. 2016c. “A Two-Step Semiparametric Method to Accommodate Sampling Weights in Multiple Imputation.” Biometrics 72: 242–252. Doi: https://10.1111/biom.12413.
Zhu, M. 2014. “Analyzing Multilevel Models with the GLIMMIX Procedure.” Paper SAS026-2014. Cary, NC: SAS Institute, Inc.