Face-to-Face or Sequential Mixed-Mode Surveys Among Non-Western Minorities in the Netherlands: The Effect of Different Survey Designs on the Possibility of Nonresponse Bias

Open access


This article compares the quality of response samples based on a single mode CAPI survey design with the quality of response samples based on a sequential mixed-mode (CAWI-CATICAPI) survey design among four non-Western minority ethnic groups in the Netherlands. The quality is assessed with respect to the representativity of the response samples and the estimated potential for nonresponse bias in survey estimates based on auxiliary variables and the response rate. This article also investigates if these designs systematically enhance response rates differently among various sociodemographic subgroups based on auxiliary variables. Also, costs and cost-related issues particular to this sequential mixed-mode design are discussed. The results show that sequential mixed mode surveys among non-Western ethnic minorities in the Netherlands lead to less representative response samples and show more potential for nonresponse bias in survey estimates. Furthermore, the designs lead to systematic differences in response rates among various sociodemographic subgroups, such as older age groups. Both designs also cause some of the same sociodemographic subgroups to be systematically underrepresented among all non-Western ethnic minority groups. Finally, the results show that in this instance the cost savings did not outweigh the reduction in quality.

AAPOR 2011. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. (7th edition). The American Association for Public Opinion Research. Available at: http://www.aapor.org/AM/Template.cfm?Section=Standard _Definitions2& Template=/CM/ContentDisplay.cfm& ContentID=3156 (accessed December 2013).

Andridge, R.R. and R.J. Little. 2011. “Proxy Pattern-Mixture Analysis for Survey Nonresponse.” Journal of Official Statistics 27: 153-180.

Bethlehem, J., F. Cobben, and B. Schouten. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, New Jersey: John Wiley & Sons, Inc.

Biemer, P.P. and L.E. Lyberg. 2003. Introduction to Survey Quality. Hoboken, New Jersey: John Wiley & Sons, Inc.

Bijl, R. and A. Verweij. 2012. Measuring and Monitoring Immigrant Integration in Europe: Integration Policies and Monitoring Efforts in 17 European Countries, 2012-8.

Den Haag: SCP. Available at: http://www.scp.nl/english/Publications/ Publications_by_year/Publications_2012/Measuring_and_monitoring_immigrant_ integration_in_Europe (accessed June 2013).

CBS-Statline. Available at: http://statline.cbs.nl/StatWeb/publication/?VW= T& DM=SLNL& PA=37325& D1=a& D2=0& D3=0& D4=0& D5=0-4,137,152,220,237 & D6=0,4,9, (l-1),l& HD=130605-0936& HDR=G2,G1,G3,T& STB=G4,G5 (accessed December 2013).

Couper, P. 2005. “Technology Trends in Survey Data Collection.” Social Science Computer Review 23: 486-501. DOI: http://dx.doi.org/10.1177/0894439305278972.

Dagevos, J. and R. Schellingerhout. 2003. “Sociaal-culturele integratie. Contacten, cultuur en orientatie op de eigen groep.” In Rapportage minderheden, edited by J. Dagevos, M. Gijsberts, and v. C. Praag, 317-362. [In Dutch: Socio-Cultural integration. Contacts, culture and focus on the own ethnic group]. Den Haag: SCP. Available at: http://www. scp.nl/Publicaties/Alle_publicaties/Publicaties_2003/Rapportage_minderheden_2003 (accessed December 2012).

De Leeuw, E.D. 2005. “To Mix or not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics 21: 233-255.

De Leeuw, E.D., D.A. Dillman, and J.J. Hox. 2008. “Mixed-Mode Surveys: When and Why.” In International Handbook Of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 299-316. New York: Taylor and Francis Group.

Dempster, A.P., N.M. Laird, and D.B. Rubin. 1977. “Maximum Likelihood from Incomplete Data via the EMAlgorithm.” Journal of the Royal Statistical Society. SeriesB (Methodological) 39: 1-38.

Dillman, D.A. 2007. Mail and Internet Surveys: The Tailored Design Method (2nd ed.). New York: John Wiley & Sons Inc.

Dillman, D.A. and L.M. Christian. 2005. “Survey Mode as a Source of Instability in Responses Across Surveys.” Field Methods 17: 30-52. DOI: http://dx.doi.org/10.1177/ 1525822X04269550.

Feskens, R.C.W. 2009. Difficult Groups in Survey Research and the Development of Tailor-made Approach Strategies. Den Haag/Utrecht: Statistics Netherlands and Kappelhof: Face-to-Face or Sequential Mixed-Mode 27 Utrecht University. Available at: http://www.cbs.nl/NR/rdonlyres/8F317AA9-1074- 4BF7-84EB-015D013DBB80/0/2009x11feskenspub.pdf (accessed October 2011).

Feskens, R.C.W., J. Kappelhof, J. Dagevos, and I.A.L Stoop. 2010. Minderheden in de mixed-mode? Een inventarisatie van voor- en nadelen van het inzetten van verschillende dataverzamelingsmethoden onder niet-westerse migranten. SCP-special 57. [In Dutch: Ethnic minorities in the mixed mode? An inventory of the advantages and disadvantages of employing different data collection methods among non-Western migrant] Den Haag: SCP. Available at: http://www.scp.nl/Publicaties/Alle_publicaties/ Publicaties_2010/Minderheden_in_de_mixed_mode (accessed December 2014).

Gijsberts, M. and J. Iedema. 2011. “Opleidingsniveau van niet-schoolgaanden en leerprestaties in het basisonderwijs.” In Jaarrapport Integratie 2011, edited by M. Gijsberts, W. Huijnk, and J. Dagevos, 76-99. [In Dutch: Education level of persons who do not attend school and educational attainment in primary school]. Den Haag: SCP. (2012-3). Available at: http://www.scp.nl/Publicaties/Alle_publicaties/ Publicaties_2012/Jaarrapport_integratie_2011 (accessed December 2014).

Graham, J.W., A.E. Olchowski, and T.D. Gilreath. 2007. “How Many Imputations are Really Needed? Some Practical Clarifications of Multiple Imputation Theory.” Prevention Science 8: 206-213. DOI: http://dx.doi.org/10.1007/s11121-007-0070-9.

Groves, R.M. 1989. Survey Costs and Survey Errors. New York: Wiley.

Groves, R.M. and M.P. Couper. 1998. Nonresponse in Household Interview Surveys. New York: Wiley. Groves, R.M. and E. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias.” Public Opinion Quarterly 72: 167-189. DOI: http://dx.doi.org/10.1093/poq/ nfn011.

Kappelhof, J.W.S. 2010. Op maat gemaakt? Een evaluatie van enkele respons verbeterende maatregelen onder Nederlanders van niet-westerse afkomst. [In Dutch: An evaluation of several response enhancing measures among Dutch of non-Western origin]. Den Haag: SCP. (SCP-special 53). Available at: http://www.scp.nl/Publicaties/ Alle_publicaties/Publicaties_2010/Op_maat_gemaakt (accessed December 2014).

Kappelhof, J.W.S. 2014. “The Effect of Different Survey Designs on Nonresponse in Surveys Among Non-Western Minorities in The Netherlands.” Survey Research Methods 8: 81-98. Available at: http://www.surveymethods.org (accessed December 2014).

Kemper, F. 1998. “Gezocht: Marokkanen. Methodische problemen bij het verwerven en interviewen van allochtone respondenten.” [In Dutch: Wanted: Moroccans. Methodological problems with obtaining response and interviewing respondents of foreign origin]. Migrantenstudies 1: 43-57.

Korte, K. and J. Dagevos. 2011. Survey Integratie Minderheden 2011. Verantwoording van de opzet en uitvoering van een survey onder Turkse, Marokkaanse, Surinaamse en Antilliaanse Nederlanders en een autochtone vergelijkingsgroep. [In Dutch: Survey on the Integration of Minorities 2011. Report on the design and implementation of the survey among Turkish, Moroccan, Surinamese, Antillean Dutch and a mainstream Dutch reference group]. Den Haag: SCP.

Kreuter, F. 2013. Improving Surveys with Paradata. Analytic Uses of Process Information. Hoboken, New Jersey: John Wiley & Sons, Inc. 28 Journal of Official Statistics Little, R.J. and D.B. Rubin. 2002. Statistical Analysis With Missing Data. New York: John Wiley & Sons.

Maitland, A., C. Casas-Cordero, and F. Kreuter. 2009. “An Evaluation of Nonresponse Bias Using Paradata from a Health Survey.” In: Proceedings of the Section on Government Statistics: American Statistical Association, Joint Statistical Meetings, 2009. 370-378. Alexandria, VA: American Statistical Association. Available at: http:// www.amstat.org/sections/SRMS/proceedings/y2009/Files/303004.pdf (accessed December 2014).

Moorman, P.G., B. Newman, R.C. Millikan, C.K.J. Tse, and D.P. Sandler. 1999. “Participation rates in a Case-control Study: The Impact of Age, Race, and Race of Interviewer.” Annals of epidemiology 9: 188-195. DOI: http://dx.doi.org/10.1016/ S1047-2797(98)00057-X.

Reep, C. 2003. Moeilijk Waarneembare Groepen. Een inventarisatie. [In Dutch: Hard to survey populations: An inventory]. Voorburg/Heerlen: CBS. (H1568-03-s00).

Rubin, D.B. 1987. Multiple Imputation for Nonresponse in Surveys. Hoboken, New Jersey: John Wiley & Sons, Inc.

Schmeets, H. 2005. “De leefsituatie van allochtonen.” In Enquêteonderzoek onder allochtonen. Problemen en oplossingen, edited by H. Schmeets en and R. van der Bie, 169-176. [In Dutch: The living conditions of Dutch of foreign origin]. Voorburg/Heerlen: CBS.

Schothorst, Y. 2002. “Onderzoek onder allochtonen: wat mag, wat moet en wat kan?” In: Interviewen in de multiculturele samenleving. Problemen en oplossingen, edited by H. Houtkoop en and J. Veenman, 101-116. [In Dutch: Survey research among Dutch of foreign origin: What may be done, what has to be done and what is possible?]. Assen: Koninklijke Van Gorcum.

Schouten, B. and F. Cobben. 2007. R-Indexes for the Comparison of Different Fieldwork Strategies and Data Collection Modes. Voorburg/Heerlen: CBS. (Discussion Paper 07002). Available at: http://www.risq-project.eu/papers/schouten-cobben-2007-a.pdf (accessed December 2013).

Schouten, B.F. and F. Cobben. 2008. An Empirical Validation of R-Indicators. Voorburg/Heerlen: CBS. (Discussion Paper 08006) Available at: http://www. risq-project.eu/papers/cobben-schouten-2008-a.pdf (accessed December 2013).

Schouten,B. F. Cobben, and J. Bethlehem. 2009. “Indicators of Representativeness of Survey Nonresponse.” Survey Methodology 35: 101-113. Available at: http://www.risq-project. eu/papers/schouten-cobben-bethlehem-2009.pdf (accessed December 2013).

Schouten, B. J. Bethlehem, K. Beullens, Ø. Kleven, G. Loosveldt, A. Luiten, K. Rutar, N. Shlomo, and C. Skinner. 2012. “Evaluating, Comparing, Monitoring, and Improving Representativity of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80: 382-399. Available at: http://onlinelibrary.wiley. com/doi/10.1111/j.1751-5823.2012.00189.x/abstract (accessed December 2013).

Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativity of Response.” Journal of Official Statistics 27: 231-253.

Shlomo, N., C. Skinner, B. Schouten, T. Carolina, and M. Morren. 2009. Partial Indicators for Representative Response. Rep. No. RISQ deliverable 4. version 2. Available at: http:// www.risq-project.eu/papers/RISQ-Deliverable-4-V2.pdf (accessed December 2013). Kappelhof: Face-to-Face or Sequential Mixed-Mode 29 Singer, E. 2002. “The Use of Incentives to Reduce Nonresponse in Household Surveys.” In Survey Nonresponse, edited by R.M. Groves, D. Dillman, J.L. Eltinge, and R.J.A. Little, 163-177. New York: John Wiley & Sons.

Singer, E., J. Van Hoewyk, N. Gebler, T. Raghunathan, and K. McGonagle. 1999. “The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys.” Journal of Official Statistics 15: 217-230.

Singer, E., J. Van Hoewyk, and M.P. Maher. 2000. “Experiments With Incentives in Telephone Surveys.” Public Opinion Quarterly 64: 171-188. DOI: http://dx.doi.org/ 10.1086/317761.

Smulders, M. 2011. Onderzoeksverantwoording Survey Integratie Minderheden 2011 [In Dutch: research description on the Survey on the Integration of Minorities 2011]. Dongen: GFK Panel Services Benelux B.V. Stoop, I.A.L. 2005. The Hunt for the Last Respondent: Nonresponse in Sample Surveys. Den Haag: SCP.

Särndal, C.-E. 2011. “Three Factors to Signal Non Response Bias With Applications to Categorical Auxiliary Variables.” International Statistical Review 79: 233-254. DOI: http://dx.doi.org/10.1111/j.1751-5823.2011.00142.x.

Särndal, C.-E., and S. Lundström. 2010. “Design for Estimation: Identifying Auxiliary Vectors to Reduce Nonresponse Bias.” Survey Methodology 36: 131-144.

Van Ingen, E., J. De Haan, and M. Duimel. 2007. Achterstand en afstand. Digitale vaardigheden van lager opgeleiden, ouderen, allochtonen en inactieven. [In Dutch: Lagging behind. Digital skills of lower educated, elderly, foreign origin and inactive persons]. Den Haag: SCP. (SCP- 2007/24).

Veenman, J. 2002. “Interviewen in multicultureel Nederland.” In Interviewen in de multiculturele samenleving. Problemen en oplossingen, edited by H. Houtkoop en and J.

Veenman, 1-19. [In Dutch: Interviewing in the multi-cultural Netherlands]. Assen: Koninklijke Van Gorcum.

Voogt, R.J.J. and W.E. Saris. 2005. “Mixed Mode Designs: Finding the Balance Between Nonresponse Bias and Mode Effects.” Journal of Official Statistics 21: 367-387.

Wagner, J. 2008. Adaptive Survey Design to Reduce Nonresponse Bias, Michigan: University of Michigan. Available at: http://deepblue.lib.umich.edu/bitstream/2027.42/ 60831/1/jameswag_1.pdf (accessed December 2014).

Wagner, J. 2010. “The Fraction of Missing Information as a Tool for Monitoring the Quality of Survey Data.” Public Opinion Quarterly 74: 223-243. DOI: http://dx.doi.org/10.1093/poq/nfq007.

Journal of Official Statistics

The Journal of Statistics Sweden

Journal Information

IMPACT FACTOR 2017: 0.662
5-year IMPACT FACTOR: 1.113

CiteScore 2016: 0.63

SCImago Journal Rank (SJR) 2016: 0.710
Source Normalized Impact per Paper (SNIP) 2016: 0.975


All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 38 38 19
PDF Downloads 8 8 5