Dropout Rates and Response Times of an Occupation Search Tree in a Web Survey

Open access


Occupation is key in socioeconomic research. As in other survey modes, most web surveys use an open-ended question for occupation, though the absence of interviewers elicits unidentifiable or aggregated responses. Unlike other modes, web surveys can use a search tree with an occupation database. They are hardly ever used, but this may change due to technical advancements. This article evaluates a three-step search tree with 1,700 occupational titles, used in the 2010 multilingual WageIndicator web survey for UK, Belgium and Netherlands (22,990 observations). Dropout rates are high; in Step 1 due to unemployed respondents judging the question not to be adequate, and in Step 3 due to search tree item length. Median response times are substantial due to search tree item length, dropout in the next step and invalid occupations ticked. Overall the validity of the occupation data is rather good, 1.7-7.5% of the respondents completing the search tree have ticked an invalid occupation.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Armstrong J.S. Denniston Jr. W.B. and Gordon M.M. (1975). The Use of the Decomposition Principle in Making Judgments. Organizational Behavior and Human Performance 14 257-263. DOI: http://www.dx.doi.org/10.1016/ 0030-5073(75)90028-8

  • Blumenstiel J.E. Roßmann J. and Steinbrecher M. (2010). Breakoff in Web Surveys of the German Longitudinal Election Study (GLES). Paper presented at the General Online Research Conference (GOR) 2010. Available at: http://www.websm.org/db/ 12/13906/Bibliography/Breakoff_in_Web_Surveys_of_the_German_Longitudinal_ Election_Study_GLES/ (accessed November 1 2013).

  • Callegaro M. Yang Y. Villar A. (2011) Should We Use the Progress Bar in Online Surveys? A Meta-Analysis of Experiments Manipulating Progress Indicators. Mannheim: Paper presented at the General Online Research Conference (GOR). Available at: http://conftool.gor.de/conftool11/index.php?page¼browseSessions&presentations¼hide&form_session¼4 (accessed November 1 2013).

  • Conrad F.G. and Couper M.P. (2008) Classifying Open Occupation Descriptions in the Current Population Survey. Ann Arbor MI: Paper presented at the conference on Optimal Coding of Open-Ended Survey Data University of Michigan.). Available at: http://www.electionstudies.org/conferences/2008Methods/ConradCouper.pdf (accessed November 1 2013).

  • De Vries J. and Ganzeboom H.B.G. (2008). Hoe meet ik beroep? Open en gesloten vragen naar beroep toegepast in statusverwervingsonderzoek. Mens & Maatschappij 83 70-95.

  • Elias P. (1997). Occupational Classification (ISCO-88). Concepts Methods Reliability Validity and Cross-National Comparability. Paris: OECD Labour Market and Social Policy Occasional Papers No. 20. DOI: http://www.dx.doi.org/10.1787/304441717388

  • Elias P. and Birch M. (2006). The Review of ISCO-88: A European Perspective. Warwick: University of Warwick Working paper Institute for Employment Research.

  • Elias P. and McKnight A. (2001). Skill Measurement in Official Statistics: Recent Developments in the UK and the Rest of Europe. Oxford Economic Papers 3 508-540. DOI: http://www.dx.doi.org/10.1093/oep/53.3.508

  • European Union (2009). Commission Regulation (EC) No. 1022/ of 29 October amending Regulations (EC) No. 1738/2005 (EC) No. 698/2006 and (EC) No. 377/2008 as regards the International Standard Classification of Occupations (ISCO). Official Journal of the European Union L 283/3 30 October. Available at: http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri¼OJ:L:2009:283:0003:0004:EN:PDF (accessed November 1 2013).

  • Eurostat (2009). Task Force on the Quality of the Labour Force Survey. Final report. Luxembourg: Publications Office of the European Union. Available at: http://epp.eurostat. ec.europa.eu/cache/ITY_OFFPUB/KS-RA-09-020/EN/KS-RA-09-020-EN.PDF (accessed November 1 2013).

  • Fazio R.H. (1990). A Practical Guide to the Use of Response Latency in Social Psychological Research. In Research methods in personality and social research C. Hendrick and M.S. Clark (eds). Newbury: Sage.

  • Funke F. and Reips U.D. (2007). Dynamics Form: Online Surveys 2.0. Leipzig: Presentation at the General Online Research conference 26-28 March 2007. Available at: http://frederikfunke.net/papers/pdf/Funke&Reips(2007)Slides_Dynamic_Forms.pdf (accessed November 1 2013).

  • Galesic M. (2006). Dropouts on the Web: Effects of Interest and Burden Experienced During an Online Survey. Journal of Official Statistics 22 313-328.

  • Ganzeboom H.B.G. (2010). Occupation Coding using ISCO-08. Training session for PIAAC Bologna January 20th. Amsterdam: Free University Amsterdam. Available at: http://www.harryganzeboom.nl/Pdf/2010-Ganzeboom-ISCO08-PIAAC-Bologna-(presentation).pdf (accessed November 1 2013).

  • Greenwood A.M. (2004). Updating the International Standard Classification of Occupations ISCO-08. Geneva: ILO Bureau of Statistics. Available at: http://millenniumindicators.un.org/unsd/class/intercop/training/escwa04/escwa04-9.PDF (accessed November 1 2013).

  • Heerwegh D. and Loosveldt G. (2006). Survey Length Statements Progress Indicators and Survey Sponsor Logos. Journal of Official Statistics 22 191-210.

  • Hunter D. (2009). ISCO-08 Draft definitions. Geneva: ILO. Available at: http://www.ilo.org/public/english/bureau/stat/isco/isco08/index.htm (accessed November 1 2013).

  • Kaczmirek L. (2009). Human-Survey Interaction: Usability and Nonresponse in Online Surveys. Neue Schriften zur Online-Forschung 6. Cologne: Halem.

  • Lenzner T. Kaczmirek L. and Lenzner A. (2010). Cognitive Burden of Survey Questions and Response Times: A Psycholinguistic Experiment. Applied Cognitive Psychology 24 1003-1020. DOI: http://www.dx.doi.org/10.1002/acp.1602

  • Malhotra N. (2008). Completion Time and Response Order Effects in Web Surveys. Public Opinion Quarterly 72 914-934. DOI: http://www.dx.doi.org/10.1093/poq/nfn050

  • Reips U.-D. (2002). Standards for Internet-Based Experimenting. Experimental Psychology 49 243-256 DOI: http://www.dx.doi.org/10.1027/1618-3169.49.4.243

  • Steinmetz S. Tijdens K.G. Bianchi A. and Biffignandi S. (2014). Improving Web Survey Quality - Potentials and Constraints of Propensity Score Weighting. Online panel research: A Data Quality Perspective ed. M. Callegaro R.P. Baker J. Bethlehem A.S. Go¨ ritz J.A. Krosnick and P.J. Lavraskas. Chichester: Wiley (forthcoming).

  • Tippins N.T. and Hilton M.L. (Eds) (2010). A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Panel to Review the Occupational Information Network (O*NET). Washington (DC): The National Academies Press. Available at: http://www.nap.edu/catalog/12814.html (accessed November 1 2013).

  • Tijdens K.G. Van Zijl S. Hughie-Williams M. Van Klaveren M. and Steinmetz S. (2010). Codebook and Explanatory Note on the WageIndicator Dataset a Worldwide Continuous Multilingual Web-Survey on Work and Wages with Paper Supplements. Amsterdam: University of Amsterdam AIAS Working Paper 102. Available at: http://www.uva-ias.net/uploaded_files/publications/WP102-TijdensvanZijlHughie-WilliamsvKlaverenSteinmetz.pdf (accessed November 1 2013).

  • Tijdens K.G. (2010). Measuring Occupations in Web-Surveys the Database. Amsterdam: University of Amsterdam AIAS Working Paper 86. Available at: http://www.uva-aias. net/uploaded_files/publications/WP86-Tijdens.pdf (accessed November 1 2013).

  • Tourangeau R. Rips L.J. and Rasinski K. (2000). The Psychology of Survey Response. Cambridge: Cambridge University Press.

  • Van der Laan P. and Van Nunspeet W. (2009). Modernising Household Surveys in the Netherlands: Design Efficiency Gains and Perspectives. Paper prepared for the European Directors of Social Statistics Seminar “Re-engineering of Social Statistics: A Perspective” Luxembourg 25 September 2009. The Hague/Heerlen: Statistics Netherlands Discussion paper 09044. Available at: http://www.cbs.nl/NR/rdonlyres/0D30D23B-FE40-4570-B41A-E9B2CADF01DB/0/200944x10pub.pdf (accessed November 1 2013).

  • Yan T. and Tourangeau R. (2008). Fast Times and Easy Questions: The Effects of Age Experience and Question Complexity on Web Survey Response Times. Applied Cognitive Psychology 22 51-68. DOI: http://www.dx.doi.org/10.1002/acp.1331

Journal information
Impact Factor

IMPACT FACTOR 2018: 0.837
5-year IMPACT FACTOR: 0.934

CiteScore 2018: 1.04

SCImago Journal Rank (SJR) 2018: 0.963
Source Normalized Impact per Paper (SNIP) 2018: 1.020

Cited By
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 365 193 8
PDF Downloads 184 114 6