Interviewer Effects on a Network-Size Filter Question

Open access

Abstract

There is evidence that survey interviewers may be tempted to manipulate answers to filter questions in a way that minimizes the number of follow-up questions. This becomes relevant when ego-centered network data are collected. The reported network size has a huge impact on interview duration if multiple questions on each alter are triggered. We analyze interviewer effects on a network-size question in the mixed-mode survey “Panel Study ‘Labour Market and Social Security’” (PASS), where interviewers could skip up to 15 follow-up questions by generating small networks. Applying multilevel models, we find almost no interviewer effects in CATI mode, where interviewers are paid by the hour and frequently supervised. In CAPI, however, where interviewers are paid by case and no close supervision is possible, we find strong interviewer effects on network size. As the area-specific network size is known from telephone mode, where allocation to interviewers is random, interviewer and area effects can be separated. Furthermore, a difference-in-difference analysis reveals the negative effect of introducing the follow-up questions in Wave 3 on CAPI network size. Attempting to explain interviewer effects we neither find significant main effects of experience within a wave, nor significantly different slopes between interviewers.

American Association for Public Opinion Research (AAPOR). 2003. Interviewer Falsification in Survey Research: Current Best Methods for Prevention, Detection, and Repair of its Effects. Available at: https://www.aapor.org/AAPOR_Main/media/MainSiteFiles/falsification.pdf (accessed April 6th, 2016).

Bethmann, A. and D. Gebhardt. 2011. User Guide “Panel Study Labor Market and Social Security” (PASS) * Wave 3. FDZ Datenreport, 04/2011 (en), Nuremberg. Available at: http://doku.iab.de/fdz/reporte/2011/DR_04-11_EN.pdf (accessed April 6th, 2016).

Biemer, P.P. and S.L. Stokes. 1989. “The Optimal Design of Quality Control Samples to Detect Interviewer Cheating.” Journal of Official Statistics 5: 23–39.

Biemer, P.P. 2010. “Overview of Design Issues: Total Survey Error.” In Handbook of Survey Research, 2nd ed., edited by P.V. Marsden and J.D. Wright, 27–58. Bingley: Emerald.

Blasius, J. and J. Friedrichs. 2013. “Faked Interviews.” In Methods, Theories, and Empirical Applications in the Social Sciences, edited by S. Salzborn, E. Davidov, and J. Reinecke, 49–56. Wiesbaden: VS Verlag für Sozialwissenschaften.

Brashears, M.E. 2011. “Small Networks and High Isolation? A Reexamination of American Discussion Networks.” Social Networks 33: 331–341. Doi: http://dx.doi.org/10.1016/j.socnet.2011.10.003.

Bredl, S., P. Winker, and K. Koetschau. 2012. “A Statistical Approach to Detect Interviewer Falsification of Survey Data.” Survey Methodology 38: 1–10.

Brüderl, J., B. Huyer-May, and C. Schmiedeberg. 2013. “Interviewer Behavior and the Quality of Social Network Data.” In Interviewers’ Deviations in Surveys. Impact, Reasons, Detection and Prevention, edited by P. Winkler, R. Porst, and N. Menold, 147–160. Frankfurt: Peter Lang.

Büngeler, K., M. Gensicke, J. Hartmann, R. Jäckle, and N. Tschersich. 2010. IAB-Haushaltspanel im Niedrigeinkommensbereich Welle 3 (2008/09): Methoden- und Feldbericht. FDZ Methodenreport, 10/2010 (de), Nuremberg. Available at: http://doku.iab.de/fdz/reporte/2010/MR_10-10.pdf (accessed April 6th, 2016).

Campanelli, P. and C. O’Muircheartaigh. 1999. “Interviewers, Interviewer Continuity, and Panel Survey Nonresponse.” Quality & Quantity 33: 59–76. Doi: http://dx.doi.org/10.1023/A:1004357711258.

Cleveland, W.S. 1979. “Robust Locally Weighted Regression and Smoothing Scatterplot.” Journal of the American Statistical Association 74: 829–836.

Cragg, J.G. 1971. “Some Statistical Models for Limited Dependent Variables with Application to the Demand for Durable Goods.” Econometrica 39: 829–844. Doi: http://dx.doi.org/10.2307/1909582.

Crespi, L.P. 1945. “The Cheater Problem in Polling.” Public Opinion Quarterly 9: 431–445.

Eckman, S., F. Kreuter, A. Jäckle, A. Kirchner, S. Presser, and R. Tourangeau. 2014. “Assessing the Mechanisms of Misreporting to Filter Questions in Surveys.” Public Opinion Quarterly 78: 721–733. Doi: http://dx.doi.org/10.1093/poq/nfu030.

Fischer, C.S. 2009. “The 2004 GSS Finding of Shrunken Social Networks: An Artifact?” American Sociological Review 74: 657–669. Doi: http://dx.doi.org/10.1177/000312240907400408.

Freeman, J. and E.W. Butler. 1976. “Some Sources of Interviewer Variance in Surveys.” Public Opinion Quarterly 40: 79–91. Doi: http://dx.doi.org/10.1086/268-269.

Goldstein, H. 1986. “Multilevel Mixed Linear Model Analysis Using Iterative Generalized Least Squares.” Biometrika 73: 43–56. Doi: http://dx.doi.org/10.1093/biomet/73.1.43.

Groves, R.M. and N.H. Fultz. 1985. “Gender Effects among Telephone Interviewers in a Survey of Economic Attitudes.” Sociological Methods Research 14: 31–52. Doi: http://dx.doi.org/10.1177/0049124185014001002.

Groves, R.M. 1989. Survey Errors and Survey Costs. New York: Wiley.

Groves, R.M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2004. Survey Methodology. Hoboken, NJ: Wiley.

Guterbrock, T.M. 2008. “Falsifications.” In Handbook of Survey Research, edited by P.J. Lavrakas, 267–270. Los Angeles: Sage.

Huddy, L., J. Billig, J. Bracciodieta, L. Hoeffler, P.J. Moynihan, and P. Pugliani. 1997. “The Effect of Interviewer Gender on the Survey Response.” Political Behavior 19: 197–220. http://dx.doi.org/10.1023/A:1024882714254.

Hughes, A., J. Chromy, K. Giacoletti, and D. Odom. 2002. “Impact of Interviewer Experience on Respondent Reports of Substance Use.” In Redesigning an Ongoing National Household Survey, edited by J. Gfroerer, J. Eyerman, and J. Chromy, 161–184. Washington: Substance Abuse and Mental Health Services Administration.

Kosyakova, Y., J. Skopek, and S. Eckman. 2015. “Do Interviewers Juggle Filter Questions? Evidence from a Multilevel Approach”. International Journal of Public Opinion Research 27: 417–431. Doi: http://dx.doi.org/10.1093/ijpor/edu027.

Kreuter, F., S. McCulloch, S. Presser, and R. Tourangeau. 2011. “The Effects of Asking Filter Questions in Interleafed Versus Grouped Format.” Sociological Methods & Research 40: 88–104. Doi: http://dx.doi.org/10.1177/0049124110392342.

Lechner, M. 2011. “The Estimation of Causal Effects by Difference-In-Difference Methods.” Foundations and Trends in Econometrics 4: 165–224.

Longford, N.T. 1993. Random Coefficient Models. Oxford: Oxford University Press.

Mangione, T.W., F.J. Fowler, and T.A. Louis. 1992. “Question Characteristics and Interviewer Effects.” Journal of Official Statistics 8: 293–307.

Marsden, P.V. 2003. “Interviewer Effects in Measuring Network Size Using a Single Name-Generator.” Social Networks 25: 1–16. Doi: http://dx.doi.org/10.1016/S0378-8733(02)00009-6.

Matschinger, H., S. Bernert, and M.C. Angermeyer. 2005. “An Analysis of Interviewer Effects on Screening Questions in a Computer Assisted Personal Mental Health Interview.” Journal of Official Statistics 21: 657–674.

McPherson, M., L. Smith-Lovin, and M.E. Brashears. 2006. “Social Isolation in America: Changes in Core Discussion Networks over Two Decades.” American Sociological Review 71: 353–375. Doi: http://dx.doi.org/10.1177/000312240607100301.

Nübling, M., H.H. Andersen, A. Mühlbacher, J. Schupp, and G.G. Wagner. 2007. “Computation of Standard Values for Physical and Mental Health Scale Scores Using the SOEP Version of SF12v2.” Schmollers Jahrbuch: Journal of Applied Social Science Studies 127: 171–182. Available at: https://www.researchgate.net/publication/23645941_Computation_of_Standard_Values_for_Physical_and_Mental_Health_Scale_Scores_Using_the_SOEP_Version_of_SF12v2 (accessed April 6th, 2016).

O’Connell, A.A. 2006. Logistic Regression Models for Ordinal Response Variables. Thousand Oaks, CA: Sage.

O’Muircheartaigh, C. and P. Campanelli. 1998. “The Relative Impact of Interviewer Effects and Sample Design Effects on Survey Precision.” Journal of the Royal Statistical Society. Series A (Statistics in Society) 161: 63–77. Doi: http://dx.doi.org/10.1111/1467-985X.00090.

Paik, A. and K. Sanchagrin. 2013. “Social Isolation in America: An Artifact.” American Sociological Review 78: 339–360. Doi: http://dx.doi.org/10.1177/0003122413482919.

Pinheiro, J.C. and D.M. Bates. 1995. “Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model.” Journal of Computational and Graphical Statistics 4: 12–35. Doi: http://dx.doi.org/10.1080/10618600.1995.10474663.

Pinheiro, J.C. and E.C. Chao. 2006. “Efficient Laplacian and Adaptive Gaussian Quadrature Algorithms for Multilevel Generalized Linear Mixed Models.” Journal of Computational and Graphical Statistics 15: 58–81. Doi: http://dx.doi.org/10.1198/106186006X96962.

Putnam, R.D. 1995. “Bowling Alone: America’s Declining Social Capital.” Journal of Democracy 6: 65–78. Doi: http://dx.doi.org/10.1353/jod.1995.0002.

Rabe-Hesketh, S. and A. Skrondal. 2012. Multilevel and Longitudinal Modeling Using Stata. Volume II:Categorical Responses, Counts, and Survival. College Station, TX: Stata Press.

Rasbash, J. and H. Goldstein. 1994. “Efficient Analysis of Mixed Hierarchical and Cross-Classified Random Structures Using a Multilevel Model.” Journal of Educational and Behavioral Statistics 19: 337–350. Doi: http://dx.doi.org/10.3102/10769986019004337.

Schnell, R. 1991. “Der Einfluß gefälschter Interviews auf Survey-Ergebnisse.” Zeitschrift für Soziologie 20: 25–35.

Schnell, R. 2012. Survey-Interviews: Methoden standardisierter Befragungen. Wiesbaden: VS Verlag.

Schnell, R. and F. Kreuter. 2000. “Untersuchungen zur Ursache unterschiedlicher Ergebnisse sehr ähnlicher Viktimisierungssurveys.” Kölner Zeitschrift für Soziologie und Sozialpsychologie 52: 96–117. Doi: http://dx.doi.org/10.1007/s11577-000-0005-y.

Schnell, R. and F. Kreuter. 2005. “Separating Interviewer and Sampling-Point Effects.” Journal of Official Statistics 21: 389–410.

Schraepler, J.P. and G.G. Wagner. 2005. “Characteristics and Impact of Faked Interviews in Surveys – An Analysis of Genuine Fakes in the Raw Data of SOEP.” Allgemeines Statistisches Archiv 89: 7–20. Doi: http://dx.doi.org/10.1007/s101820500188.

Schuman, H. and J. Converse. 1971. “The Effects of Black and White Interviewers on Black Responses in 1968.” Public Opinion Quarterly 35: 44–68. Doi: http://dx.doi.org/10.1086/267866.

Snijders, T.A.B. and R. Bosker. 2000. Multilevel Analysis. London: Sage.

StataCorp. 2011. Stata. Longitudinal-Data/Panel-Data Reference Manual. Release 12. College Station, TX: StataCorp.

Swamy, P.A.V.B. 1971. Statistical Inference in a Random Coefficient Model. New York: Springer.

Tourangeau, R. and T. Yan. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin 133: 859–883.

Tourangeau, R., F. Kreuter, and S. Eckman. 2012. “Motivated Underreporting in Screening Interviews.” Public Opinion Quarterly 76: 453–469. Doi: http://dx.doi.org/10.1093/poq/nfs033.

Tourangeau, R., F. Kreuter, and S. Eckman. 2013. Motivated Misreporting: Shaping Answers to Reduce Survey Burden. In Survey Measurement: Techniques and Findings from Recent Research, edited by U. Engel, 24–41, Frankfurt: Campus.

Trappmann, M., S. Gundert, C. Wenzig, and D. Gebhardt. 2010. “PASS: a Household Panel Survey for Research on Unemployment and Poverty.” Schmollers Jahrbuch. Journal of Applied Social Science Studies 130: 609–622 Doi: http://dx.doi.org/10.3790/schm.130.4.609.

Trappmann, M., J. Beste, A. Bethmann, and G. Mü ller. 2013. “The PASS Panel Survey After Six Waves.” Journal for Labour Market Research 46: 275–281. Doi: http://dx.doi.org/10.1007/s12651-013-0150-1.

van der Zouwen, J., W. Dijkstra, and J.H. Smit. 2004. “Studying Respondent-Interviewer Interaction: The Relationship Between Interviewing Style, Interviewer Behavior, and Response Behavior.” In Measurement Errors in Surveys, edited by P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz, and S. Sudman, 419–437. New York: Wiley.

van Tilburg, T.G. 1998. “Interviewer Effects in the Measurement of Personal Network Size. A Non-Experimental Study.” Sociological Methods and Research 26: 300–328. Doi: http://dx.doi.org/10.1177/0049124198026003002.

Vassallo, R., G.B. Durrant, and P.W.F. Smith. 2016. Separating Interviewer and Area Effects Using a Cross-Classified Multilevel Logistic Model: Simulation Findings and Implications for Survey Designs. Submitted manuscript (available from the author on request: ).

West, B.T., F. Kreuter, and U. Jaenichen. 2013. “Interviewer Effects in Face-to-Face Surveys: A Function of Sampling, Measurement Error, or Nonresponse?” Journal of Official Statistics 29: 277–297. Doi: http://dx.doi.org/10.2478/jos-2013-0023.

Journal of Official Statistics

The Journal of Statistics Sweden

Journal Information


IMPACT FACTOR 2017: 0.662
5-year IMPACT FACTOR: 1.113

CiteScore 2017: 0.74

SCImago Journal Rank (SJR) 2017: 1.158
Source Normalized Impact per Paper (SNIP) 2017: 0.860

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 81 81 40
PDF Downloads 22 22 9