Web-Face-to-Face Mixed-Mode Design in a Longitudinal Survey: Effects on Participation Rates, Sample Composition, and Costs

Open access

Abstract

Sequential mixed-mode designs are increasingly considered as an alternative to interviewer-administered data collection, allowing researchers to take advantage of the benefits of each mode. We assess the effects of the introduction of a sequential web-face-to-face mixed-mode design over three waves of a longitudinal survey in which members were previously interviewed face-to-face. Findings are reported from a large-scale randomised experiment carried out on the UK Household Longitudinal Study. No differences are found between the mixed-mode design and face-to-face design in terms of cumulative response rates and only minimal differences in terms of sample composition. On the other hand, potential cost savings are evident.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Bianchi A. and S. Biffignandi. 2014. “Responsive Design for Economic Data in Mixed-Mode Panels.” In Contribution to Sampling Statistics edited by F. Mecatti P.L. Conti and M.G. Ranalli 85–102. Springer International Publishing.

  • Bianchi A. and S. Biffignandi. 2017. “Representativeness in Panel Surveys.” To appear in Mathematical Population Studies.

  • Bianchi A. and S. Biffignandi. Forthcoming. “Survey Experiments on Interactions: a Case Study of Incentives and Modes.” In Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment edited by P.J. Lavrakas E. de Leeuw A. Holbrook C. Kennedy M.W. Traugott and B.T. West. Hoboken NJ: John Wiley & Sons.

  • Biemer P.P. 2010. “Total Survey Error: Design Implementation and Evaluation.” Public Opinion Quarterly 74(5): 817–848. Doi: http://dx.doi.org/10.1093/poq/nfq058.

  • Biemer P.P. and L.E. Lyberg. 2003. Introduction to Survey Quality. New York: John Wiley.

  • Bowling A. 2005. “Mode of Questionnaire Administration Can Have Serious Effects on Data Quality.” Journal of Public Health 27: 281–291.

  • Buck N. and S. McFall. 2012. “Understanding Society: Design Overview.” Longitudinal and Life Course Studies 3: 5–17.

  • Calinescu M. and B. Schouten. 2015. “Adaptive Survey Designs to Minimize Survey Mode Effects – a Case Study on the Dutch Labor Force Survey.” Survey Methodology 41(2) : 403–425.

  • Chng S. M. White C. Abraham and S. Skippon. 2016. “Commuting and Wellbeing in London: the Roles of Commute Mode and Local Public Transport Connectivity.” Preventive Medicine 88: 182–188. Doi: http://dx.doi.org/10.1016/j.ypmed.2016.04.014.

  • Couper M. 2011. “The Future of Modes of Data Collection.” Public Opinion Quarterly 75: 889–908. Doi: http://dx.doi.org/10.1093/poq/nfr046.

  • Evandrou M. J. Falkingham Z. Feng and A. Vlachantoni. 2016. “Ethnic Inequalities in Limiting Health and Self-Reported Health in Later Life Revisited.” Journal of Epidemiology & Community Health 70: 653–662. Doi: http://dx.doi.org/10.1136/jech-2015-206074.

  • Fan W. and Z. Yan. 2010. “Factors Affecting Response Rates of the Web Surveys: a Systematic Review.” Computers in Human Behavior 26: 132–139.

  • Farrant G. and C. O’Muircheartaigh. 1991. “Components of Nonresponse Bias in the British Election Surveys.” In Understanding Political Change edited by A. Heath J. Curtice R. Jowell S. Evans J. Field and S. Witherspoon 235–249. London: Pergamon Press.

  • Fong B. and J. Williams. 2011. “British Crime Survey: Feasibility of Boosting Police Force Area (PFA) Sample Sizes Using Supplementary Recontact Surveys.” Report for the Home Office TNS-BMRB London.

  • Gaia A. 2014. “Does a Mixed-Mode Design Increase Panel Attrition? Evidence from the UKHLS Innovation Panel.” Paper presented at the Internet Survey Methodology Workshop Bolzano December 1–3.

  • Göritz A. 2006. “Incentives in Web Studies: Methodological Issues and a Review.” International Journal of Internet Science 1: 58–70.

  • Göritz A. 2010. “Using Lotteries Loyalty Points and Other Incentives to Increase Participant Response and Completion.” In Advanced methods for conducting online behavioural research edited by S. Gosling and J. Johnson 219–233. Washington DC: American Psychological Association. Doi: http://dx.doi.org/10.1037/12076-014.

  • Göritz A. 2015. “Incentive Effects.” In Improving Survey Methods: Lessons from Recent Research edited by U. Engel B. Jann P. Lynn A. Scherpenzeel and P. Sturgis 339–350. London: Routledge.

  • Griffin D. D. Fischer and M. Morgan. 2001. “Testing an Internet Response Option for the American Community Survey.” Paper presented at the annual conference of the American Association for Public Opinion Research Montreal.” Quebec Canada May 17–20.

  • Groves R. and L. Lyberg. 2010. “Total Survey Error: Past Present and Future.” Public Opinion Quarterly 74(5): 849–879. Doi: http://dx.doi.org/10.1093/poq/nfq065.

  • Groves R.M. and F. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly 72: 167–189. Doi: https://doi.org/10.1093/poq/nfn011.

  • Jäckle A. 2016. “Mode Effects on Measurement in Understanding Society.” Paper presented at the International Panel Survey Methods Workshop Berlin June 20–21. Available at: http://www.diw.de/en/diw_01.c.534396.en/program_psmw2016.html (accessed March 2017).

  • Jäckle A. and P. Lynn. 2008. “Respondent Incentives in a Multi-Mode Panel Survey: Cumulative Effects on Nonresponse and Bias.” Survey Methodology 34: 105–117.

  • Jäckle A. P. Lynn and J. Burton. 2015. “Going Online with a Face-to-Face Household Panel: Effects of a Mixed Mode Design on Item and Unit Non-Response.” Survey Research Methods 9(1): 57–70. Doi: http://dx.doi.org/10.18148/srm/2015.v9i1.5475.

  • Janssen B. 2006. “Web Data Collection in a Mixed Mode Approach: An Experiment.” Paper presented at the European Conference on Quality in Official Statistics (Q2006) Cardiff April 24–26. Available at: webarchive.nationalarchives.gov.uk/20140721132900/http://ons.gov.uk/about/newsroom/events/q2006––european-conference-on-quality-in-survey-statistics-24-26-april-2006/agenda/session-19-wednesday.pdf (accessed March 2017).

  • Klausch T. J. Hox and B. Schouten. 2015a. “Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population.” Journal of the Royal Statistical Society Series A 178(4): 945–961. Doi: http://dx.doi.org/10.1111/rssa.12102.

  • Klausch T. B. Schouten and J.J. Hox. 2015b. “Evaluating Bias of Sequential Mixedmode Designs Against Benchmark Surveys.” Sociological Methods & Research : 1–34. Doi: http://dx.doi.org/10.1177/0049124115585362.

  • Kreuter F. 2013. “Facing the Nonresponse Challenge.” The Annals of the American Academy of Political and Social Science 645: 23–35. Doi: https://doi.org/10.1177/0002716212456815.

  • Krosnick J.A. and D.F. Alwin. 1987. “An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement.” Public Opinion Quarterly 51: 201–219. Doi: https://doi.org/10.1086/269029.

  • Lagerstrøm B. 2008. “Cost Efficiency in a Mixed-Mode Survey – Evidence from the Norwegian Rent Market Survey.” Paper presented at the 19th International Workshop on Household Survey Nonresponse Ljubljana September 15. Available at: http://www.nonresponse.org/db/3/558/Biblliography/Cost%20efficiency%20in%20a%20mixed-mode%20survey%20Evidence%20from%20the%20Norwegian%20Rent%20Marked%20Survey/?&p1=308&p2=74&p3=551 (accessed March 2017).

  • Laurie H. and P. Lynn. 2009. “The Use of Respondent Incentives on Longitudinal Surveys.” In Methodology of Longitudinal Surveys edited by P. Lynn 205–233. Chichester: Wiley.

  • Leesti T. 2010. “Canadian Labour Force Survey Internet Data Collection Pilot Test.” Paper presented at the Fifth Workshop on Labour Force Survey Methodology Paris April 15–16.

  • De Leeuw E.D. 2005. “To Mix or not to Mix Data Collection in Surveys.” Journal of Official Statistics 21: 233–255.

  • Lugtig P. 2014. “Panel Attrition: Separating Stayers Fast Attriters Gradual Attriters and Lurkers.” Sociological Methods and Research 43(4): 699–723. Doi: http://dx.doi.org/10.1177/0049124113520305.

  • Lynn P. 2009. “Sample Design for Understanding Society.” Understanding Society Working Paper 2009-01 ISER University of Essex Colchester. Available at: www.understandingsociety.ac.uk/research/publications/working-paper/understandingsociety/2009-01 (accessed 16 March 2017).

  • Lynn P. 2013. “Alternative Sequential Mixed-Mode Designs: Effects on Attrition Rates Attrition Bias and Costs.” Journal of Survey Statistics and Methodology 1: 183–205. Doi: http://dx.doi.org/10.1093/jssam/smt015.

  • Lynn P. 2014. “Targeted Response Inducement Strategies on Longitudinal Surveys.” In Improving Survey Methods: Lessons from Recent Research edited by U. Engel B. Jann P. Lynn A. Scherpenzeel and P. Sturgis. Routledge/Psychology Press.

  • Lynn P. Forthcoming. “Tackling Panel Attrition.” In The Palgrave Handbook of Survey Research edited by D.L. Vannette and J.A. Krosnick. Palgrave.

  • Lynn P. and A. Jäckle. Forthcoming. “Mounting Multiple Experiments on Longitudinal Social Surveys: Design and Implementation Considerations.” In Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment edited by P.J. Lavrakas E.E. de Leeuw A. Holbrook C. Kennedy M.W. Traugott and B.T. West. Hoboken NJ: Wiley.

  • Lynn P. and P. Lugtig. 2017. “Total Survey Error for Longitudinal Surveys.” In Total Survey Error in Practice edited by Paul Biemer Edith de Leeuw Stephanie Eckman Brad Edwards Frauke Kreuter Lars Lyberg Clyde Tucker and Brady West 279–298. Hoboken New Jersey: Wiley.

  • Lynn P. and D. Lievesley. 1991. “Drawing General Population Samples in Great Britain.” London: SCPR.

  • Lynn P. S.C.N. Uhrig and J. Burton. 2010. “Lessons from a Randomized Experiment with Mixed-Mode Designs for a Household Panel Survey.” Understanding Society Working Paper Series 2010-03.

  • Martin P. and P. Lynn. 2011. “The Effects of Mixed Mode Survey Designs on Simple and Complex Analyses.” ISER Working Paper Series 2011-28. Colchester: Institute for Social and Economic Research University of Essex. Available at: https://www.iser.essex.ac.uk/research/publications/working-papers/iser/2011-28 (accessed February 5 2013).

  • Schoeni R. F. Stafford K. McGonagle and P. Andreski. 2013. “Response Rates in National Panel Surveys.” The Annals of the American Academy of Political and Social Science 645: 60–87. Doi: https://doi.org/10.1177/0002716212456363.

  • Schwarz N. F. Strack H.-J. Hippler and G. Bishop. 1991. “The Impact of Administration Mode on Response Effects in Survey Measurement.” Applied Cognitive Psychology 5: 193–212.

  • Souren M. 2012. “Multi-Mode Surveys at Statistics Netherlands: Implications Experiences and Open Issues.” Paper presented at Opening Conference of the European Statistical System Network (ESSNet) on Data Collection for Social Surveys using Multiple Modes Wiesbaden October 11–12 2012.

  • Uhrig S.C.N. 2008. “The Nature and Causes of Attrition in the British Household Panel Study.” Institute for Social and Economic Research Working Paper 2008-05. Available at: https://www.iser.essex.ac.uk/research/publications/working-papers/iser/2008-05 (accessed 16 March 2017).

  • Uhrig S.C.N. 2011. “Using Experiments to Guide Decision Making in Understanding Society: Introducing the Innovation Panel.” In Understanding Society: Early Findings from the First Wave of the UK’s Household Longitudinal Study edited by S.L. McFall and C. Garrington. Colchester: University of Essex. Available at: http://research.understandingsociety.org.uk/findings/early-findings (accessed 16 March 2017).

  • Voogt R. and W. Saris. 2005. “Mixed Mode Designs: Finding the Balance between Nonresponse Bias and Mode Effects.” Journal of Official Statistics 21(3): 367–387.

  • Voorpostel M. and V.A. Ryser. 2011. “Mixed Mode Data Collection as a Strategy to Decrease Panel Attrition in the Swiss Household Panel.” FORS Working Paper 2_11. Available at: http://ohs-shp.unil.ch/workingpapers/WP2_11.pdf (accessed 28 March 2017).

  • Wallace S. J. Nazroo and L. Bécares. 2016. “Cumulative Effect of Racial Discrimination on the Mental Health of Ethnic Minorities in the United Kingdom.” American Journal of Public Health 106(7): 1294–1300. Doi: http://dx.doi.org/10.2105/AJPH.2016.303121.

  • Watson N. and M. Wooden. 2014. “Re-Engaging with Survey Non-Respondents: Evidence from Three Household Panels.” Journal of the Royal Statistical Society Series A (Statistics in Society) 177(2): 499–522. Doi: http://dx.doi.org/10.1111/rssa.12024.

Search
Journal information
Impact Factor


IMPACT FACTOR 2018: 0,837
5-year IMPACT FACTOR: 0,934

CiteScore 2018: 1.04

SCImago Journal Rank (SJR) 2018: 0.963
Source Normalized Impact per Paper (SNIP) 2018: 1.020

Cited By
Metrics
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 429 300 4
PDF Downloads 265 209 4