“Interviewer” Effects in Face-to-Face Surveys: A Function of Sampling, Measurement Error, or Nonresponse?

Open access


Recent research has attempted to examine the proportion of interviewer variance that is due to interviewers systematically varying in their success in obtaining cooperation from respondents with varying characteristics (i.e., nonresponse error variance), rather than variance among interviewers in systematic measurement difficulties (i.e., measurement error variance) - that is, whether correlated responses within interviewers arise due to variance among interviewers in the pools of respondents recruited, or variance in interviewer-specific mean response biases. Unfortunately, work to date has only considered data from a CATI survey, and thus suffers from two limitations: Interviewer effects are commonly much smaller in CATI surveys, and, more importantly, sample units are often contacted by several CATI interviewers before a final outcome (response or final refusal) is achieved. The latter introduces difficulties in assigning nonrespondents to interviewers, and thus interviewer variance components are only estimable under strong assumptions. This study aims to replicate this initial work, analyzing data from a national CAPI survey in Germany where CAPI interviewers were responsible for working a fixed subset of cases.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Bauer T.K. Fertig M. and Vorrell M. (2011a). Neighborhood Effects and Individual Unemployment. SOEP paper 409 DIW Berlin.

  • Bauer T.K. Flake R. and Sinning M.G. (2011b). Labor Market Effects of Immigration: Evidence from Neighborhood Data. Ruhr Economic Papers #257 Bochum Dortmund Duisburg Essen.

  • Beland Y. and St-Pierre M. (2008). Mode Effects in the Canadian Community Health Survey: A Comparison of CATI and CAPI. Advances in Telephone Survey Methodology Chapter 14 J.M. Lepkowski C. Tucker J.M. Brick E.D. de Leeuw L. Japec P.J. Lavrakas M.W. Link R.L. Sangsler (eds). New York: Wiley.

  • Bender S. and Heining J. (2011). The Research-Data-Centre in Research-Data-Centre Approach: A First Step Towards Decentralised International Data Sharing. FDZ Methodenreport 07/2011 Nürnberg.

  • Biemer P.P. (1980). A Survey Error Model which Includes Edit and Imputation Error. Proceedings of the American Statistical Association Section on Survey Research Methods 616-621.

  • Biemer P.P. (2010). Total Survey Error: Design Implementation and Evaluation. Public Opinion Quarterly 74 817-848.

  • Biemer P.P. and Stokes S.L. (1991). Approaches to the Modeling of Measurement Error. Measurement Errors in Surveys P.P. Biemer R.M. Groves L. Lyberg N.A. Mathiowetz and S. Sudman (eds). New York: Wiley. Biemer P.P. and Trewin D. (1997). A Review of Measurement Error Effects on the Analysis of Survey Data. In Survey Measurement and Process Quality L. Lyberg P.P.

  • Biemer M. Collins E.D. de Leeuw C. Dippo N. Schwarz and D. Trewin (eds). New York: Wiley-Interscience.

  • Brunton-Smith I. Sturgis P. and Williams J. (2012). Is Success in Obtaining Contact and Cooperation Correlated with the Magnitude of Interviewer Variance? Public Opinion Quarterly 76 265-286.

  • Büngeler K. Gensicke M. Hartmann J. Jäckle R. and Tschersich N. (2010). IABHaushaltspanel im Niedrigeinkommensbereich Welle 3 (2008/2009). Methoden- und Feldbericht. FDZ Methodenreport 10/2010 Nu¨rnberg.

  • Campanelli P. and O’Muircheartaigh C. (1999). Interviewers Interviewer Continuity and Panel Survey Nonresponse. Journal of Official Statistics 2 303-314.

  • Christoph B. Müller G. Gebhardt D. Wenzig C. Trappmann M. Achatz J. Tisch A. and Gayer C. (2008). Codebook and Documentation of the Panel Study “Labour Market and Social Security” (PASS). Volume 1: Introduction and Overview Wave 1 (2006/2007). FDZ Datenreport 05/2008 Nürnberg.

  • Collins M. and Butcher B. (1982). Interviewer and Clustering Effects in an Attitude Survey. Journal of the Market Research Society 25 39-58.

  • Davis P. and Scott A. (1995). The Effect of Interviewer Variance on Domain Comparisons. Survey Methodology 21 99-106.

  • Durrant G.B. Groves R.M. Staetsky L. and Steele F. (2010). Effects of Interviewer Attitudes and Behaviors on Refusal in Household Surveys. Public Opinion Quarterly 74 1-36.

  • Fellegi I.P. (1964). Response Variance and its Estimation. Journal of the American Statistical Association 59 1016-1041.

  • Freeman J. and Butler E.W. (1976). Some Sources of Interviewer Variance in Surveys. Public Opinion Quarterly 40 79-91.

  • Gebhardt D. Müller G. Bethmann A. Trappmann M. Christoph B. Gayer C. Müller B. Tisch A. Siflinger B. Kiesl H. Huyer-May B. Achatz J. Wenzig C. Rudolph H. Graf T. and Biedermann A. (2009). Codebook and Documentation of the Panel Study “Labour Market and Social Security” (PASS). Volume 1: Introduction and Overview Wave 2 (2007/2008). FDZ Datenreport 06/2009 Nürnberg.

  • Groves R.M. (2004). The Interviewer as a Source of Survey Measurement Error. Survey Errors and Survey Costs (Second Edition). New York: Wiley-Interscience.

  • Groves R.M. and Lyberg L. (2010). Total Survey Error: Past Present and Future. Public Opinion Quarterly 74 849-879.

  • Groves R.M. and Magilavy L.J. (1984). An Experimental Measurement of Total Survey Error. Proceedings of the Joint Statistical Meetings of the American Statistical Association Section on Survey Research Methods 698-703.

  • Groves R.M. and Magilavy L.J. (1986). Measuring and Explaining Interviewer Effects in Centralized Telephone Surveys. Public Opinion Quarterly 50 251-266.

  • Groves R.M. and Peytcheva E. (2008). The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis. Public Opinion Quarterly 72 167-189.

  • Hansen M.H. Hurwitz W.N. and Bershad M.A. (1960). Measurement Errors in Censuses and Surveys. Bulletin of the International Statistical Institute 32nd Session Vol. 38 Part 2 359-374.

  • Holbrook A.L. Green M.C. and Krosnick J.A. (2003). Telephone Versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Bias. Public Opinion Quarterly 67 79-125.

  • Hox J.J. (1998). Multilevel Modeling: When and Why. In Classification Data Analysis and Data Highways I. Balderjahn R. Mathar and M. Schader (eds). New York: Springer.

  • Hox J.J. and de Leeuw E.D. (2002). The Influence of Interviewers’ Attitude and Behavior on Household Survey Nonresponse: An International Comparison. Survey Nonresponse R.M. Groves D.A. Dillman J.L. Eltinge and R.J.A. Little (eds). New York: Wiley.

  • Kish L. (1962). Studies of Interviewer Variance for Attitudinal Variables. Journal of the American Statistical Association 57 92-115.

  • Mangione T.W. Fowler F.J. and Louis T.A. (1992). Question Characteristics and Interviewer Effects. Journal of Official Statistics 8 293-307.

  • Morton-Williams J. (1993). Interviewer Approaches. Aldershot: Dartmouth Publishing Company Limited.

  • O’Muircheartaigh C. and Campanelli P. (1998). The Relative Impact of Interviewer Effects and Sample Design Effects on Survey Precision. Journal of the Royal Statistical Society Series A 161 63-77.

  • O’Muircheartaigh C. and Campanelli P. (1999). A Multilevel Exploration of the Role of Interviewers in Survey Non-Response. Journal of the Royal Statistical Society Series A 162(Part 3) 437-446.

  • Patterson H.D. and Thompson R. (1971). Recovery of Inter-Block Information when Block Sizes are Unequal. Biometrika 58 545-554.

  • Pickery J. and Loosveldt G. (2002). A Multilevel Multinomial Analysis of Interviewer Effects on Various Components of Unit Nonresponse. Quality and Quantity 36 427-437.

  • Rudolph H. and Trappmann M. (2007). Design und Stichprobe des Panels “Arbeitsmarkt und Soziale Sicherung” (PASS). In Neue Daten für die Sozialstaatsforschung. Zur Konzeption der IAB-Panelerhebung “Arbeitsmarkt und Soziale Sicherung” (ed.) M. Promberger IAB Forschungsbericht 12/2007 Nürnberg.

  • Schaeffer N.C. Dykema J. and Maynard D.W. (2010). Interviewers and Interviewing. In Handbook of Survey Research J.D. Wright and P.V. Marsden (eds). (Second Edition). Bingley U.K. Emerald Group Publishing Limited.

  • Schnell R. (1997). Nonresponse in Bevölkerungsumfragen: Ausmaß Entwicklung und Ursachen. Opladen: Leske þ Budrich.

  • Schnell R. (2012). Survey-Interviews. Standardisierte Befragungen in den Sozialwissenschaften. Wiesbaden: VS-Verlag.

  • Schnell R. and Kreuter F. (2005). Separating Interviewer and Sampling-Point Effects. Journal of Official Statistics 21 389-410.

  • Snijkers G. Hox J.J. and de Leeuw E.D. (1999). Interviewers’ Tactics for Fighting Survey Nonresponse. Journal of Official Statistics 15 185-198.

  • Trappmann M. Gundert S. Wenzig C. and Gebhardt D. (2010). PASS: a Household Panel Survey for Research on Unemployment and Poverty. Proceedings of the American Statistical Association Section on Survey Research Methods 130 609-622.

  • West B.T. and Olson K. (2010). How Much of Interviewer Variance is Really Nonresponse Error Variance? Public Opinion Quarterly 74 1004-1026.

  • Wiggins R.D. Longford N.T. and O’Muircheartaigh C.A. (1992). A Variance Components Approach to Interviewer Effects. In Survey and Statistical Computing A. Westlake R. Banks C. Payne and T. Orchard (eds). Amsterdam: North-Holland.

  • Zhang D. and Lin X. (2008). Variance Component Testing in Generalized Linear Mixed Models for Longitudinal/Clustered Data and Other Related Topics. Random Effect and Latent Variable Model Selection D.B. Dunson (ed.). Springer Lecture Notes in Statistics 192.

Journal information
Impact Factor

IMPACT FACTOR 2018: 0,837
5-year IMPACT FACTOR: 0,934

CiteScore 2018: 1.04

SCImago Journal Rank (SJR) 2018: 0.963
Source Normalized Impact per Paper (SNIP) 2018: 1.020

Cited By
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1745 1194 29
PDF Downloads 614 497 8