Mixed mode data collection designs are increasingly being adopted with the hope that they may reduce selection errors in single mode survey designs. Yet possible reductions in selection errors achieved by mixing modes may be offset by a potential increase in total survey error due to extra measurement error being introduced by the additional mode(s). Few studies have investigated this empirically, however. In the present study, we compute the Mean Squared Error (MSE) for a range of estimates using data from a mode comparison experiment. We compare two mixed mode designs (a sequential web plus mail survey, and a combined concurrent and sequential CATI plus mail survey) with a single mode mail survey. The availability of auxiliary data on the sampling frame allows us to estimate several components of MSE (sampling variance, non-coverage, nonresponse and measurement bias) for a number of sociodemographic and target variables. Overall, MSEs are lowest for the single mode survey, and highest for the CATI plus mail design, though this pattern is not consistent across all estimates. Mixing modes generally reduces total bias, but the relative contribution to total survey error from different sources varies by design and by variable type.
Brick, J.M. and D. Williams. 2013. “Explaining Rising Nonresponse Rates in Cross-Sectional Surveys.” The ANNALS of the American Academy of Political and Social Science 645: 36–59. Doi: http://dx.doi.org/10.1177/0002716212456834.
Chang, L. and J.A. Krosnick. 2009. “National surveys via RDD telephone interviewing versus the internet. Comparing sample representativeness and response quality.” Public Opinion Quarterly 73: 641–678. Doi: http://dx.doi.org/10.1093/poq/nfp075.
De Leeuw, E. 2005. “To Mix or Not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics 21: 233–255.
De Leeuw, E.D. and N. Berzelak. 2017. “Survey mode or survey modes?” In The Sage Handbook of Survey Methodology, edited by C. Wolf, D. Joye, T.W. Smith, and Y.-C. Fu, 142–156. London: Sage Publications.
De Leeuw, E. and W. de Heer. 2002. “Trends in Household Survey Nonresponse: A Longitudinal and International Comparison.” In Survey nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.
De Leeuw, E.D., J.J. Hox, and D.A. Dillman. 2008. “Mixed mode surveys: When and why.” In International Handbook of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 299–316. New York/London: Erlbaum/Taylor & Francis.
Dillman, D.A., G. Phelps, R. Tortora, K. Swift, J. Kohrell, J. Berck, and B.L. Messer. 2009. “Response Rate and Measurement Differences in Mixed-Mode Surveys using Mail, Telephone, Interactive Voice Response (IVR) and the Internet.” Social Science Research 38: 1–18. Doi: http://dx.doi.org/10.1016/j.ssresearch.2008.03.007.
Dillman, D.A., J.D. Smyth, and L.M. Christian. 2014. Internet, Phone, Mail, and Mixedmode Surveys: the Tailored Design Method (4th Edition). Hoboken: Wiley.
Eva, G., G. Loosveldt, P. Lynn, P. Martin, M. Revilla, W. Saris, and J. Vannieuwenhuyze. 2010. Assessing the Cost-Effectiveness of Different Modes for ESS Data Collection. London: City University.
Fowler, F.J., P.M. Gallagher, V.L. Stringfellow, A.M. Zaslavsky, J.W. Thompson, and P.D. Cleary. 2002. “Using Telephone Interviews to Reduce Nonresponse Bias to Mail Surveys of Health Plan Members.” Medical Care 40: 190–200.
Gordoni, G., P. Schmidt, and Y. Gordoni. 2012. “Measurement invariance across face-to-face and telephone modes: the case of minority-status collectivistic oriented groups.” International Journal of Public Opinion Research 24(2): 185–207. Doi: http://dx.doi.org/10.1093/ijpor/edq054.
Groves, R.M., F.J. Fowler Jr., M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2009. Survey Methodology (Wiley Series in Survey Methods), 2nd Ed. Hoboken, NJ: John Wiley & Sons.
Groves, R.M. and L.J. Magilavy. 1984. “An Experimental Measurement of Total Survey Error.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 698–703. Alexandria, VA: American Statistical Association. Available at: http://ww2.amstat.org/sections/srms/Proceedings/ (accessed March 2017).
Heerwegh, D. and G. Loosveldt. 2011. “Assessing mode effects in a national crime victimization survey using structural equation models: social desirability bias and acquiescence.” Journal of Official Statistics 27: 49–63.
Hochstim, J.R. 1967. “A Critical Comparison of Three Strategies of Collecting Data from Households.” Journal of the American Statistical Association 62: 976–989. Doi: http://dx.doi.org/10.2307/2283686.
Holbrook, A., M. Green, and J. Krosnick. 2003. “Telephone Versus Face-to-face Interviewing of National Probability Samples with Long Questionnaires.” Public Opinion Quarterly 67: 79–125. Doi: http://dx.doi.org/10.1086/346010.
Hox, J., E.D. de Leeuw, and T. Klausch. 2017. “Mixed mode research: Issues in design and analysis.” In Total Survey Error in Practice: Improving Quality in the Era of Big Data, edited by P.P. Biemer, E.D. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, C. Tucker, and B.T. West, 511–530. Hoboken, NJ: John Wiley and Sons, Inc.
Hox, J.J., E.D. de Leeuw, and E.A.O. Zijlmans. 2015. “Measurement equivalence in mixed mode surveys.” Frontiers in Psychology: Quantitative Psychology and Measurement. Doi: http://dx.doi.org/10.3389/fpsyg.2015.00087.
Kappelhof, J.W.S. 2013. “The Effect of Different Survey Designs on Nonresponse in Surveys of Non-Western Minorities in The Netherlands.” Survey Research Methods 8(2): 81–98. Doi: http://dx.doi.org/10.18148/srm/2014.v8i2.5784.
Klausch, L.T., J.J. Hox, and B. Schouten. 2013. “Measurement effects of survey mode on the equivalence of attitudinal rating scale questions.” Sociological Methods and Research 42: 227–263. Doi: http://dx.doi.org/10.1177/0049124113500480.
Klausch, T., J.J. Hox, and B. Schouten. 2015a. “Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population.” Journal of the Royal Statistical Society Series A 178: 945–961. Doi: http://dx.doi.org/10.1111/rssa.12102.
Klausch, T., B. Schouten, and J.J. Hox. 2015b. “Evaluating Bias of Sequential Mixedmode Designs Against Benchmark Surveys.” Sociological Methods and Research 1–34. Doi: http://dx.doi.org/10.1177/0049124115585362.
Kreuter, F., S. Presser, and R. Tourangeau. 2008. “Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity.” Public Opinion Quarterly 72: 847–865. Doi: http://dx.doi.org/10.1093/poq/nfn063.
Kreuter, F., G. Müller, and M. Trappmann. 2010. “Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data.” Public Opinion Quarterly 74: 880–906. Doi: http://dx.doi.org/10.1093/poq/nfq0.
Link, M.W. and A.H. Mokdad. 2006. “Can Web and Mail Survey Modes Improve Participation in an RDD-Based National Health Surveillance?” Journal of Official Statistics 22: 293–312.
Lynn, P. 2013. “Alternative Sequential Mixed-Mode Designs: Effects on Attrition Rates, Attrition Bias, and Costs.” Journal of Survey Statistics and Methodology 1: 183–205. Doi: http://dx.doi.org/10.1093/jssam/smt015.
Massey, D.S. and R. Tourangeau. 2013. “Where Do We Go from Here? Nonresponse and Social Measurement.” The Annals of the American Academy of Political and Social Science 645(1): 222–236. Doi: http://dx.doi.org/10.1177/0002716212464191.
Olson, K. 2006. “Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias.” Public Opinion Quarterly 70(5): 737–758. Doi: http://dx.doi.org/10.1093/poq/nfl038.
Olson, K., J.D. Smyth, and H. Wood. 2012. “Does Providing Sample Members with Their Preferred Survey Mode Really Increase Participation Rates?” Public Opinion Quarterly 76(4): 611–635. Doi: http://dx.doi.org/10.1093/poq/nfs024.
Peytchev, A., R.K. Baxter, and L.R. Carley-Baxter. 2009. “Not All Survey Effort is Equal. Reduction of Nonresponse Bias and Nonresponse Error.” Public Opinion Quarterly 73(4): 785–806. Doi: http://dx.doi.org/10.1093/poq/nfp037.
Peytcheva, E. and R.M. Groves. 2009. “Using Variation in Response Rates of Demographic Subgroups as Evidence of Nonresponse Bias in Survey Estimates.” Journal of Official Statistics 25(2): 193–201.
Roberts, C., D. Joye, M. Ernst Stähli, and R. Sanchez Tome. 2016. Mixing modes of data collection in Swiss social surveys: Methodological Report of the LIVES-FORS Mixed Mode Experiment. LIVES Working Paper Series, 2016/48. Doi: http://dx.doi.org/10.12682/lives.2296-1658.2016.48.
Sakshaug, J.W., T. Yan, and R. Tourangeau. 2010. “Nonresponse Error, Measurement Error, and Mode of Data Collection: Tradeoffs in a Multi-mode Survey of Sensitive and Non-sensitive Items.” Public Opinion Quarterly 74(5): 907–933. Doi: http://dx.doi.org/10.1093/poq/nfq057.
Schouten, B., J. van den Brakel, B. Buelens, J. van der Laan, and T. Klausch. 2013. “Disentangling Mode-Specific Selection and Measurement Bias in Social Surveys.” Social Science Research 42(6): 1555–1570. Doi: http://dx.doi.org/10.1016/j.ssresearch.2013.07.005.
Siemiatycki, J. 1979. “A Comparison of Mail, Telephone, and Home Interview Strategies for Household Health Surveys.” American Journal of Public Health 69: 238–245.
Tourangeau, R. 2017. “Mixing modes: Tradeoffs among coverage, nonresponse, and measurement error.” In Total Survey Error in Practice: Improving Quality in the Era of Big Data, edited by P.P. Biemer, E.D. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, C. Tucker, and B.T. West. 115–132. Hoboken, NJ: John Wiley and Sons, Inc.
Tourangeau, R., R.M. Groves, and C.D. Redline. 2010. “Sensitive topics and reluctant respondents: Demonstrating a link between nonresponse bias and measurement error.” Public Opinion Quarterly 74(3): 413–432. Doi: http://dx.doi.org/10.1093/poq/nfq004.
Vannieuwenhuyze, J.T.A., G. Loosveldt, and G. Molenberghs. 2010. “A Method for Evaluating Mode Effects in Mixed-mode Surveys.” Public Opinion Quarterly 74: 1027–1045. Doi: http://dx.doi.org/10.1093/poq/nfq059.
Vannieuwenhuyze, J.T.A. and G. Loosveldt. 2012. “Evaluating Relative Mode Effects in Mixed-mode Surveys: Three Methods to Disentangle Selection and Measurement Effects.” Sociological Methods and Research 42: 82–104. Doi: http://dx.doi.org/10.1177/0049124112464868.
Wagner, J., J. Arrieta, H. Guyer, and M.B. Ofstedal. 2014. “Does Sequence Matter in Multimode Surveys: Results from an Experiment.” Field Methods 26(2): 141–155. Doi: http://dx.doi.org/10.1177/1525822X13491863.