Reaching Hard-to-Survey Populations: Mode Choice and Mode Preference

Open access


This study assesses the effect of response-mode choices on response rates, and responsemode preferences of hard-to-survey populations: young adults, full-time workers, big city inhabitants, and non-Western immigrants. Using address-based sampling, a stratified sample of 3,496 households was selected. The first group of sample members was contacted face to face and could choose between a CAPI and web response mode. The second group, contacted by telephone, could choose between CATI and web. The third group, contacted by telephone, was randomly allocated to a response mode. Our address-based sampling technique was successful in reaching most of the hard-to-survey groups. Insufficient numbers of non- Western immigrants were reached; therefore this group was excluded from our analyses. In our mixed-effect models, no significant effects on the willingness to participate were found for mode choice. We found that full-time workers and young adults were significantly more likely to choose web over CAPI when contacted face to face.

Ament, P. (2008). Most People on Long-Term Low Incomes Live in Major Cities. Statistics Netherlands. Available at: (accessed August 2013).

American Association for Public Opinion Research (2011). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Ann Arbor, MI: AAPOR.

Bates, D. (2005). Fitting Linear Mixed Models in R. R News, 5, 27-30.

Bethlehem, J., Cobben, F., and Schouten, B. (2011). Handbook of Nonresponse in Household Surveys. Hoboken, NJ: Wiley.

Blohm, M. and Diehl, C. (2001). Wenn Migranten Migranten befragen Zum Teilnahmeverhalten von Einwanderern bei Bevo¨lkerungsbefragungen. Zeitschrift fu¨ r Soziologie, 30, 223-242.

Blumberg, S.J. and Luke, J.V. (2007). Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults. Public Opinion Quarterly, 71, 734-749. DOI:

Brady, S.E., Stapleton, C.N., Bouffard, J.A., and Imel, J.D. (2003). Effect of Alternative Data Collection Modes on Cooperation Rates and Data Quality. Proceedings of the American Statistical Association, Joint Statistical Meetings, Section on Survey Research Methods, 693-700, San Francisco, August 3-7, 2003, sections/srms/Proceedings/ (accessed November 2013).

Brøgger, J., Nystad, W., Cappelen, I., and Bakke, P. (2007). No Increase in Response Rate by Adding a Web Response Option to a Postal Population Survey: A Randomized Trial. Journal of Medical Internet Research, 9(5), e40. DOI: jmir.9.5.e40

Campanelli, P., Sturgis, P., and Purdon, S. (1997). Can you Hear Me Knocking: an Investigation into the Impact of Interviewers on Survey Response Rates. London: The Survey Methods Centre SCPR.

Conrad, F.G., Schober, M.F., Zhang, C., Yan, H.G., Vickers, L., Johnston, M., Hupp, A., Hemingway, L., Fail, S., Ehlen, P., and Antoun, C. (2013). Mode Choice on an iPhone Increases Survey Data Quality. Paper presented at the Annual Conference of the American Association for Public Opinion Research, Boston., May 16-19, 2013.

Couper, M.P. and Groves, R.M. (1996). Social Environmental Impacts on Survey Cooperation. Quality & Quantity, 30, 173-188. DOI: BF00153986

Couper, M.P., Kapteyn, A., Schonlau, M., and Winter, J. (2007). Noncoverage and Nonresponse in an Internet Survey. Social Science Research, 36, 131-148. DOI:

De Gregorio, J. and Lee, J.W. (2002). Education and Income Inequality: New Evidence from Cross-Country Data. Review of Income and Wealth, 48, 395-416. DOI:

De Leeuw, E.D. (2005). To Mix or Not to Mix Data Collection Modes in Surveys.Journal of Official Statistics, 21, 233-255.

De Leeuw, E.D. and Hox, J.J. (1998). Nonrespons in Surveys: Een Overzicht. Kwantitatieve Methoden, 19, 31-53.

De Leeuw, E.D. and van der Zouwen, J. (1992). Data Quality and Mode of Data Collection: Methodology and Explanatory Model. La qualite´ de l’information dans les enquetes, L. Lebart (ed.). Paris: Dunod, 11-31.

Deutskens, E., Ruyter, K., Wetzels, M., and Oosterveld, P. (2004). Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study. Marketing Letters, 15, 21-36. DOI:

Dhar, R. (1997). Consumer Preference for a No-Choice Option. Journal of Consumer Research, 24, 215-231.

Dillman, D.A. (2007). Mail and Internet Surveys: The Tailored Design Method. Hoboken, NJ: Wiley.

Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, K., Berck, J., and Messer, B.L. (2009). Response Rate and Measurement Differences in Mixed-Mode Surveys Using Mail, Telephone, Interactive Voice Response (IVR) and the Internet. Social Science Research, 38, 1-18.

Dillman, D.A., West, K.K., and Clark, J.R. (1994). Influence of an Invitation to Answer by Telephone on Response to Census Questionnaires. Public Opinion Quarterly, 58, 557-568. DOI:

Diment, K. and Garrett-Jones, S. (2007). How Demographic Characteristics Affect Mode Preference in a Postal/Web Mixed Mode Survey of Australian Researchers. Social Science Computer Review, 25, 410-417. DOI:

Durrant, G.B. and Steele, F. (2009). Multilevel Modeling of Refusal and Non-Contact in Household Surveys: Evidence from Six UK Government Surveys. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172, 361-381. DOI: http://

Feskens, R.C.W. (2009). Difficult Groups in Survey Research and the Development of Tailor-Made Approach Strategies. Utrecht: University of Utrecht.

Feskens, R.C.W., Hox, J.J., Lensvelt-Mulders, G.J.L.M., and Schmeets, J.J.G. (2007). Non-Response Among Ethnic Minorities: a Multivariate Analysis. Journal of Official Statistics, 23, 387-408.

Feskens, R.C.W., Hox, J.J., Lensvelt-Mulders, G.J.L.M., and Schmeets, J.J.G. (2006). Collecting Data Among Ethnic Minorities in an International Perspective. Field Methods, 18, 284-304. DOI:

Friese, C.R., Lee, C.S., O’Brien, S., and Crawford, S.D. (2010). Multi-Mode and Method Experiment in a Study of Nurses. Survey Practice, 3. Available at: (accessed August 2013).

Gentry, R. and Good, C. (2008). Offering Respondents a Choice of Survey Mode: Use Patterns of an Internet Response Option in a Mail Survey. Paper presented at the Annual Conference of the American Association for Public Opinion Research, New Orleans., May 15-18, 2008.

Gillian, E., Loosveldt, G., Lynn, P., Martin, P., Revilla, M., Saris, W., and Vannieuwenhuyze, J. (2010). ESS Prep6 - Mixed-Mode Experiment. Deliverable 21

Final Mode Report. Available at: Griffin, D.H., Fischer, D.P., and Morgan, M.T. (2001). Testing an Internet Response Option for the American Community Survey. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Montreal., May 17-20, 2001.

Groves, R.M. (1977). An Experimental Comparison of National Telephone and Personal.Interview Surveys. Proceedings of the Section on Social Statistics: American Statistical Association, 232-241.

Groves, R.M., Cialdini, R.B., and Couper, M.P. (1992). Understanding the Decision to Participate in a Survey. Public Opinion Quarterly, 56, 475-495. DOI:

Groves, R.M. and Couper, M.P. (1998). Nonresponse in Household Interview Surveys. New York: Wiley.

Groves, R.M., Dillman, D.A., Eltinge, J.L., and Little, R.J.A. (2002). Survey Nonresponse. New York: Wiley.

Groves, R.M. and Kahn, R.L. (1979). Surveys by Telephone: A National Comparison with Personal Interviews. New York: Academic Press.

Goyder, J. (1987). The Silent Minority. Nonsample Members on Sample Surveys. Cambridge: Polity Press.

Goyder, J., Lock, J., and McNair, T. (1992). Urbanization Effects on Survey Non- Response: A Test Within and Across Cities. Quality and Quantity, 26, 39-48. Haan, M. and Ongena, Y.P. (2014). Tailored and Targeted Designs for Hard-to-Survey Populations. In Hard to Survey Populations, R. Tourangeau et al. (eds). Cambridge: Cambridge University Press. (in press).

Hardigan, P.C., Succar, C.T., and Fleischer, J.M. (2012). An Analysis of Response Rate and Economic Costs Between Mail and Web-Based Surveys Among Practicing Dentists: A Randomized Trial. Journal of Community Health, 37, 383-394. DOI: http://

Hoffer, T., Grigorian, K., and Fesco, R. (2007). Effectiveness of Using Respondent Mode Preference Data. Paper presented at the Joint Statistical Meetings of the American Statistical Association, Salt Lake City., July 29-August 2, 2007.

Holbrook, A.L., Green, M.C., and Krosnick, J.A. (2003). Telephone vs. Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias. Public Opinion Quarterly, 67, 79-125. DOI:

Hox, J.J. and de Leeuw, E.D. (1994). A Comparison of Nonresponse in Mail, Telephone, and Face-to-Face Surveys. Quality and Quantity, 28, 329-344. DOI: http://www. Israel, G.D. (2010). Using Web-Hosted Surveys to Obtain Responses from Extension Clients: A Cautionary Tale. Journal of Extension, 48, (accessed November 2013).

Iyengar, S.S. and Lepper, M.R. (2000). When Choice Is Demotivating: Can One Desire Too Much of a Good Thing? Journal of Personality and Social Psychology, 79, 995-1006. DOI:

Kaplowitz, M.D., Hadlock, T.D., and Levine, R. (2004). A Comparison of Web and Mail Survey Response Rates. Public Opinion Quarterly, 68, 94-101. DOI: http://

Lesser, V.M., Newton, L., and Yang, D. (2010). Does Providing a Choice of Survey Modes Influence Response? Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Chicago., May 13-16, 2010.

Li, B., Lingsma, H.F., Steverberg, E.W., and Lesaffre, E. (2011). Logistic Random Effects Regression Models: A Comparison of Statistical Packages for Binary and Ordinal Outcomes. BMC Medical Research, 11, Article 77. DOI: http://www.dx.doi.org10.1186/1471-2288

Loges, W.E. and Jung, J. (2001). Exploring the Digital Divide: Internet Connectedness and Age. Communication Research, 28, 536-562. DOI:

Lynn, P. (2013). Alternative Sequential Mixed-Mode Designs: Effects on Attrition Rates, Attrition Bias, and Costs. Journal of Survey Statistics and Methodology, 1, 183-205. DOI:

Martin, P. (2011). What Makes a Good Mix? Chances and Challenges of Mixed Mode Data Collection in the ESS. Working Paper No. 02. Centre for Comparative Social Surveys, City University, London.

Medway, R.L. and Fulton, J. (2012). When More Gets You Less: a Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates. Public Opinion Quarterly, 76, 733-746. DOI:

Millar, M.M. and Dillman, D.A. (2011). Improving Response to Web and Mixed-Mode Surveys. Public Opinion Quarterly, 75, 249-269. DOI: http://www.dx.doi.org10.1093/poq/nfr003

Millar, M.M., O’Neill, A.C., and Dillman, D.A. (2009). Are Mode Preferences Real? Technical Report 09-003, Social and Economic Sciences Research Center. Pullman: Washington State University.

Miller, T.I., Kobayashi, M.M., Caldwell, E., Thurston, S., and Collett, B. (2002). Citizen Surveys on the Web: General Population Surveys of Community Opinion. Social Science Computer Review, 20, 124-136. DOI:

Nicolaas, H., Wobma, E., and Ooijevaar, J. (2010). Demografie van (Niet-Westerse) Allochtonen in Nederland. Statistics Netherlands. Available at: (accessed August 2013).

Olson, K., Smyth, J.D., and Wood, H. (2012). Does Providing Sample Members with Their Preferred Survey Mode Really Increase Participation Rates? Public Opinion Quarterly, 76, 611-635.

Porter, S.R. and Whitcomb, M.E. (2007). Mixed-Mode Contacts in Web Surveys: Paper Is Not Necessarily Better. Public Opinion Quarterly, 71, 635-648. DOI:

Quene´, H. and van den Bergh, H. (2008). Examples of Mixed-Effects Modeling With Crossed Random Effects and With Binomial Data. Journal of Memory and Language, 59, 413-425. DOI:

Radon, K., Goldberg, M., Becklake, M., Pindur, U., Hege, I., and Nowak, D. (2002).Low Acceptance of an Internet-Based Online Questionnaire by Young Adults. Epidemiology, 13, 748-749.

Raets, B. (2008). Vinex-Bewoners zijn Geen Doorsnee Stedelingen. Statistics Netherlands. Available at: (accessed August 2013).

Ryan, J.M., Corry, J.R., Attewell, R., and Smithson, M.J. (2002). A Comparison of an Electronic Version of the SF-36 General Health Questionnaire to the Standard Paper Version. Quality of Life Research, 11, 19-26. DOI: http://www.dx.doi.org10.1023/A:1014415709997

Scherpenzeel, A. and Toepoel, V. (2012). Recruiting a Probability Sample for an Online Panel. Effects of Contact Mode, Incentives and Information. Public Opinion Quarterly, 76, 470-490. DOI:

Schmuhl, P., van Duker, H., Gurley, K.L., Webster, A., and Olson, L. (2010). Reaching Emergency Medical Services Providers: Is One Survey Mode Better Than Another? Prehospital Emergency Care, 14, 361-369.

Schneider, S.J., Cantor, D., Malakhoff, L., Arieira, C., Segel, P., Nguyen, K., and Tancreto, J.G. (2005). Telephone, Internet and Paper Data Collection Modes for the Census 2000 Short Form. Journal of Official Statistics, 21, 89-101.

Schwartz, B. (2004). The Paradox of Choice: Why More Is Less. New York: Harper Perennial.

Shih, T. and Fan, X. (2007). Response Rates and Mode Preferences in Web-Mail Mixed- Mode Surveys: A Meta-Analysis. International Journal of Internet Science, 2, 59-82.

Sinclair, M., O’Toole, J., and Malawaraarachchi, M. (2012). Comparison of Response Rates and Cost-Effectiveness for a Community-Based Survey: Postal, Internet and Telephone Modes with Generic or Personalized Recruitment Approaches. BMC Medical Research Methodology, 12, Article 132. DOI:

Smyth, J.D., Dillman, D.A., Christian, L.M., and O’Neill, A.C. (2010). Using the Internet to Survey Small Towns and Communities: Limitations and Possibilities in the Early 21st Century. American Behavioral Scientist, 53, 1423-1448. DOI:

Smyth, J.D., Olson, K., and Richards, A. (2009). Are Mode Preferences Real? Paper presented at the Annual Meeting of the American Association for Public Opinion Research. Hollywood, Florida., May 14-17, 2009.

Statistics Netherlands. (2013a). Definitions. Available at: (accessed November 2013).

Statistics Netherlands. (2013b). ICT Gebruik van Huishoudens naar Huishoudkenmerken Available at: (accessed November 2013).

Statistics Netherlands. (2010). Laag en Langdurig Laag Inkomen; Particuliere Huishoudens naar Kenmerken. Available at: (accessed August 2013).

Stoop, I. (2007). No time, Too Busy: Time Strain and Survey Cooperation. In Measuring Meaningful Data in Social Research, G. Loosveldt, M. Swyngedouw, and B. Cambre´ (eds). Leuven: Acco, 301-314.

Stoop, I. (2005). The Hunt for the Last Respondent. Non-Response in Sample Surveys. The Hague: Social and Cultural Planning Agency.

Sylvester, D.E. and McGlynn, A.J. (2010). The Digital Divide, Political Participation, and Place. Social Science Computer Review, 28, 64-74. DOI: http://www.dx.doi. org/10.1177/0894439309335148

Tancreto, J.G., Zelenak, M.F., Davis, M., Ruiter, M., and Matthews, B. (2012). 2011 American Community Survey Internet Tests: Results from First Test in April 2011. Final Report. Washington, DC: US Census Bureau.

Tarnai, J. and Paxson, M.C. (2004). Survey Mode Preferences of Business Respondents. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Phoenix., May 13-16, 2004.

Toffler, A. (1971). Future Shock. United States: Bantam Books.

Tse, A. (1998). Comparing the Response Rate, Response Speed, and Response Quality of Two Methods of Sending Questionnaires: Email vs. Mail. Journal of the Market Research Society, 40, 353-361.

Turner, S., Viera, L., and Marsh, S. (2010). Offering a Web Option in a Mail Survey of Young Adults: Impact on Survey Quality. Poster presented at the Annual Meeting of the American Association for Public Opinion Research, Chicago., May 13-16, 2010.

Vannieuwenhuyze, J. (forthcoming 2014). On the Relative Advantage of Mixed-Mode Surveys. Survey Research Methods.

Vehovar, V., Batagelj, Z., Lozar Manfreda, K., and Zalatel, M. (2002). Nonresponse in Web Surveys. In Survey Nonresponse, R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little (eds). New York: John Wiley and Sons, 229-242.

Vercruyssen, A., Roose, H., Carton, A., and van de Putte, B. (2013). The Effect of Busyness on Survey Participation: Being Too Busy or Feeling Too Busy to Cooperate? International Journal of Social Research Methodology. DOI:

Weeks, M.F., Kulka, R.A., Lessler, J.T., and Whitmore, R.W. (1983). Personal versus Telephone Surveys for Collecting Household Health Data at the Local Level. American Journal of Public Health, 73, 1389-1394. DOI:

Werner, P. and Forsman, G. (2005). Mixed Mode Data Collection Using Paper and Web Questionnaires. Proceedings of the American Statistical Association, Section on Survey Research Methods, 4015-4017.

West, B.T. (2012). An Examination of the Quality and Utility of Interviewer Observations in the National Survey of Family Growth. Journal of the Royal Statistical Society: Series A (Statistics in Society), 176, 211-225. DOI: j.1467-985X.2012.01038.x

Wilkins, J.R., Hueston, W.D., MacCrawford, J., Steele, L.L., and Gerken, D.F. (1997). Mixed-Mode Survey of Female Veterinarians Yields High Response Rate.Occupational Medicine, 47, 458-462. DOI:

Ziegenfuss, J.Y., Beebe, T.J., Rey, E., Schleck, C., Locke, III, G.R., and Talley, N.J. (2010). Internet Option in a Mail Survey: More Harm Than Good? Epidemiology, 21, 585-586. DOI:

Zickuhr, K. and Smith, A. (2012). Digital Differences. Pew Internet Project Report.Washington, DC: Pew Research Center. Available at: (accessed August 2013).

Journal of Official Statistics

The Journal of Statistics Sweden

Journal Information

IMPACT FACTOR 2017: 0.662
5-year IMPACT FACTOR: 1.113

CiteScore 2017: 0.74

SCImago Journal Rank (SJR) 2017: 1.158
Source Normalized Impact per Paper (SNIP) 2017: 0.860

Cited By


All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 168 168 39
PDF Downloads 79 79 15