Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes

Open access


Mixed mode data collection designs are increasingly being adopted with the hope that they may reduce selection errors in single mode survey designs. Yet possible reductions in selection errors achieved by mixing modes may be offset by a potential increase in total survey error due to extra measurement error being introduced by the additional mode(s). Few studies have investigated this empirically, however. In the present study, we compute the Mean Squared Error (MSE) for a range of estimates using data from a mode comparison experiment. We compare two mixed mode designs (a sequential web plus mail survey, and a combined concurrent and sequential CATI plus mail survey) with a single mode mail survey. The availability of auxiliary data on the sampling frame allows us to estimate several components of MSE (sampling variance, non-coverage, nonresponse and measurement bias) for a number of sociodemographic and target variables. Overall, MSEs are lowest for the single mode survey, and highest for the CATI plus mail design, though this pattern is not consistent across all estimates. Mixing modes generally reduces total bias, but the relative contribution to total survey error from different sources varies by design and by variable type.

Biemer, P.P. 1988. “Measuring Data Quality.” In Telephone Survey Methodology, edited by R.M. Groves, P.P. Biemer, L. Lyberg, J.T. Massey, W. II, Nicholls, and J. Waksberg, 341–375. New York: Wiley.

Biemer, P.P. 2010. “Total Survey Error: Design, Implementation, and Evaluation.” Public Opinion Quarterly 74: 817–848. Doi:

Biemer, P.P. and L.E. Lyberg. 2003. Introduction to Survey Quality. Hoboken, NJ: Wiley.

Blumberg, S.J. and J.V. Luke. 2013. “Wireless Substitution: Early Release of Estimates from the National Health Interview Survey, July–December 2012.” National Center for Health Statistics. December. Available at: (accessed March 2017).

Brick, J.M. and D. Williams. 2013. “Explaining Rising Nonresponse Rates in Cross-Sectional Surveys.” The ANNALS of the American Academy of Political and Social Science 645: 36–59. Doi:

Carley-Baxter, L.R., A. Peytchev, and M.C. Black. 2010. “Comparison of Cell Phone and Landline Surveys: A Design Perspective.” Field Methods 22(1): 3–15. Doi:

Chang, L. and J.A. Krosnick. 2009. “National surveys via RDD telephone interviewing versus the internet. Comparing sample representativeness and response quality.” Public Opinion Quarterly 73: 641–678. Doi:

Couper, M.P. 2011. “The Future of Modes of Data Collection.” Public Opinion Quarterly 75(5): 889–908. Doi:

De Leeuw, E. 2005. “To Mix or Not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics 21: 233–255.

De Leeuw, E.D. and N. Berzelak. 2017. “Survey mode or survey modes?” In The Sage Handbook of Survey Methodology, edited by C. Wolf, D. Joye, T.W. Smith, and Y.-C. Fu, 142–156. London: Sage Publications.

De Leeuw, E. and W. de Heer. 2002. “Trends in Household Survey Nonresponse: A Longitudinal and International Comparison.” In Survey nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.

De Leeuw, E.D., J.J. Hox, and D.A. Dillman. 2008. “Mixed mode surveys: When and why.” In International Handbook of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 299–316. New York/London: Erlbaum/Taylor & Francis.

Dillman, D.A., G. Phelps, R. Tortora, K. Swift, J. Kohrell, J. Berck, and B.L. Messer. 2009. “Response Rate and Measurement Differences in Mixed-Mode Surveys using Mail, Telephone, Interactive Voice Response (IVR) and the Internet.” Social Science Research 38: 1–18. Doi:

Dillman, D.A., J.D. Smyth, and L.M. Christian. 2014. Internet, Phone, Mail, and Mixedmode Surveys: the Tailored Design Method (4th Edition). Hoboken: Wiley.

Eva, G., G. Loosveldt, P. Lynn, P. Martin, M. Revilla, W. Saris, and J. Vannieuwenhuyze. 2010. Assessing the Cost-Effectiveness of Different Modes for ESS Data Collection. London: City University.

Fowler, F.J., P.M. Gallagher, V.L. Stringfellow, A.M. Zaslavsky, J.W. Thompson, and P.D. Cleary. 2002. “Using Telephone Interviews to Reduce Nonresponse Bias to Mail Surveys of Health Plan Members.” Medical Care 40: 190–200.

Gordoni, G., P. Schmidt, and Y. Gordoni. 2012. “Measurement invariance across face-to-face and telephone modes: the case of minority-status collectivistic oriented groups.” International Journal of Public Opinion Research 24(2): 185–207. Doi:

Greene, J., H. Speizer, and W. Wiitala. 2008. “Telephone and Web: Mixed-Mode Challenge.” Health Services Research 43: 230–248. Doi:

Groves, R.M. 1989. Survey Errors and Survey Costs. New York: Wiley.

Groves, R.M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70: 646–675.

Groves, R.M. 2011. “Three Eras of Survey Research.” Public Opinion Quarterly 75(5): 861–971. Doi:

Groves, R.M., F.J. Fowler Jr., M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2009. Survey Methodology (Wiley Series in Survey Methods), 2nd Ed. Hoboken, NJ: John Wiley & Sons.

Groves, R.M. and L.J. Magilavy. 1984. “An Experimental Measurement of Total Survey Error.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 698–703. Alexandria, VA: American Statistical Association. Available at: (accessed March 2017).

Heerwegh, D. and G. Loosveldt. 2011. “Assessing mode effects in a national crime victimization survey using structural equation models: social desirability bias and acquiescence.” Journal of Official Statistics 27: 49–63.

Hochstim, J.R. 1967. “A Critical Comparison of Three Strategies of Collecting Data from Households.” Journal of the American Statistical Association 62: 976–989. Doi:

Holbrook, A., M. Green, and J. Krosnick. 2003. “Telephone Versus Face-to-face Interviewing of National Probability Samples with Long Questionnaires.” Public Opinion Quarterly 67: 79–125. Doi:

Hox, J., E.D. de Leeuw, and T. Klausch. 2017. “Mixed mode research: Issues in design and analysis.” In Total Survey Error in Practice: Improving Quality in the Era of Big Data, edited by P.P. Biemer, E.D. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, C. Tucker, and B.T. West, 511–530. Hoboken, NJ: John Wiley and Sons, Inc.

Hox, J.J., E.D. de Leeuw, and E.A.O. Zijlmans. 2015. “Measurement equivalence in mixed mode surveys.” Frontiers in Psychology: Quantitative Psychology and Measurement. Doi:

Kappelhof, J.W.S. 2013. “The Effect of Different Survey Designs on Nonresponse in Surveys of Non-Western Minorities in The Netherlands.” Survey Research Methods 8(2): 81–98. Doi:

Klausch, L.T., J.J. Hox, and B. Schouten. 2013. “Measurement effects of survey mode on the equivalence of attitudinal rating scale questions.” Sociological Methods and Research 42: 227–263. Doi:

Klausch, T., J.J. Hox, and B. Schouten. 2015a. “Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population.” Journal of the Royal Statistical Society Series A 178: 945–961. Doi:

Klausch, T., B. Schouten, and J.J. Hox. 2014. “The Use of Within-subject Experiments for Estimating Measurement Effects in Mixed-mode Surveys.” Statistics Netherlands Discussion Paper, 2015/06. Available at: (accessed March 2017).

Klausch, T., B. Schouten, and J.J. Hox. 2015b. “Evaluating Bias of Sequential Mixedmode Designs Against Benchmark Surveys.” Sociological Methods and Research 1–34. Doi:

Kreuter, F., S. Presser, and R. Tourangeau. 2008. “Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity.” Public Opinion Quarterly 72: 847–865. Doi:

Kreuter, F., G. Müller, and M. Trappmann. 2010. “Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data.” Public Opinion Quarterly 74: 880–906. Doi:

Link, M.W. and A.H. Mokdad. 2006. “Can Web and Mail Survey Modes Improve Participation in an RDD-Based National Health Surveillance?” Journal of Official Statistics 22: 293–312.

Lipps, O., N. Pekari, and C. Roberts. 2015. “Undercoverage and Nonresponse in a List-Sampled Telephone Election Study.” Survey Research Methods 9(2): 71–82. Doi:

Lynn, P. 2013. “Alternative Sequential Mixed-Mode Designs: Effects on Attrition Rates, Attrition Bias, and Costs.” Journal of Survey Statistics and Methodology 1: 183–205. Doi:

Massey, D.S. and R. Tourangeau. 2013. “Where Do We Go from Here? Nonresponse and Social Measurement.” The Annals of the American Academy of Political and Social Science 645(1): 222–236. Doi:

Millar, M.M. and D.A. Dillman. 2011. “Improving Response to Web and Mixed-Mode Surveys.” Public Opinion Quarterly 75(2): 249–269. Doi:

Olson, K. 2006. “Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias.” Public Opinion Quarterly 70(5): 737–758. Doi:

Olson, K., J.D. Smyth, and H. Wood. 2012. “Does Providing Sample Members with Their Preferred Survey Mode Really Increase Participation Rates?” Public Opinion Quarterly 76(4): 611–635. Doi:

Peytchev, A., R.K. Baxter, and L.R. Carley-Baxter. 2009. “Not All Survey Effort is Equal. Reduction of Nonresponse Bias and Nonresponse Error.” Public Opinion Quarterly 73(4): 785–806. Doi:

Peytcheva, E. and R.M. Groves. 2009. “Using Variation in Response Rates of Demographic Subgroups as Evidence of Nonresponse Bias in Survey Estimates.” Journal of Official Statistics 25(2): 193–201.

Rao, J.N.K. and A.J. Scott. 1987. “On simple adjustments to chi-square tests with sample survey data.” The Annals of Statistics 15(1): 385–397. Doi:

Roberts, C., D. Joye, M. Ernst Stähli, and R. Sanchez Tome. 2016. Mixing modes of data collection in Swiss social surveys: Methodological Report of the LIVES-FORS Mixed Mode Experiment. LIVES Working Paper Series, 2016/48. Doi:

Sakshaug, J.W., T. Yan, and R. Tourangeau. 2010. “Nonresponse Error, Measurement Error, and Mode of Data Collection: Tradeoffs in a Multi-mode Survey of Sensitive and Non-sensitive Items.” Public Opinion Quarterly 74(5): 907–933. Doi:

Schouten, B., J. van den Brakel, B. Buelens, J. van der Laan, and T. Klausch. 2013. “Disentangling Mode-Specific Selection and Measurement Bias in Social Surveys.” Social Science Research 42(6): 1555–1570. Doi:

Siemiatycki, J. 1979. “A Comparison of Mail, Telephone, and Home Interview Strategies for Household Health Surveys.” American Journal of Public Health 69: 238–245.

Smith, T.W. 2011. “Refining the Total Survey Error Perspective.” International Journal of Public Opinion Research 23(4): 464–484. Doi:

Suzer-Gurtekin, Z., S. Heeringa, and R. Vaillant. 2012. “Investigating the Bias of Alternative Statistical Inference Methods in Sequential Mixed-mode Surveys.” Proceedings of the JSM, Section on Survey Research Methods 4711-2. Available at: (accessed April 2016).

Tourangeau, R. 2017. “Mixing modes: Tradeoffs among coverage, nonresponse, and measurement error.” In Total Survey Error in Practice: Improving Quality in the Era of Big Data, edited by P.P. Biemer, E.D. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, C. Tucker, and B.T. West. 115–132. Hoboken, NJ: John Wiley and Sons, Inc.

Tourangeau, R., R.M. Groves, and C.D. Redline. 2010. “Sensitive topics and reluctant respondents: Demonstrating a link between nonresponse bias and measurement error.” Public Opinion Quarterly 74(3): 413–432. Doi:

Vannieuwenhuyze, J.T.A., G. Loosveldt, and G. Molenberghs. 2010. “A Method for Evaluating Mode Effects in Mixed-mode Surveys.” Public Opinion Quarterly 74: 1027–1045. Doi:

Vannieuwenhuyze, J.T.A. 2014. “On the Relative Advantage of Mixed-Mode versus Single-Mode Surveys.” Survey Research Methods 8(1): 31–42. Doi:

Vannieuwenhuyze, J.T.A. and G. Loosveldt. 2012. “Evaluating Relative Mode Effects in Mixed-mode Surveys: Three Methods to Disentangle Selection and Measurement Effects.” Sociological Methods and Research 42: 82–104. Doi:

Wagner, J., J. Arrieta, H. Guyer, and M.B. Ofstedal. 2014. “Does Sequence Matter in Multimode Surveys: Results from an Experiment.” Field Methods 26(2): 141–155. Doi:

Journal of Official Statistics

The Journal of Statistics Sweden

Journal Information

IMPACT FACTOR 2017: 0.662
5-year IMPACT FACTOR: 1.113

CiteScore 2017: 0.74

SCImago Journal Rank (SJR) 2017: 1.158
Source Normalized Impact per Paper (SNIP) 2017: 0.860


All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 127 127 48
PDF Downloads 51 51 29