Item Response Rates for Composite Variables

Open access

Abstract

Item response rates frequently serve as indicators of data quality and potential nonresponse bias. However, key variables from surveys, such as total household income or net worth, are often composite variables constructed from several underlying components. Because such composite variables do not have clearly identifiable response rates, inference on the data quality of these key measures is more difficult. This article proposes three new methods for aggregating data on response rates across questions to create a measure of item response for composite variables. To compare the three methods and illustrate how they can be used (both individually and collectively) to investigate data quality, I analyze item response for net worth in the Survey of Income and Program Participation (SIPP) and the Survey of Consumer Finances (SCF). These new measures provide detailed information about net worth estimates that would be difficult to assess without an item response aggregation method. Overall, these new item response rate methods provide a new way of describing data quality for key measures in surveys and for analyzing changes in data quality over time.

American Association for Public Opinion Research. 2016. “Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition.” AAPOR. Available at: https://www.aapor.org/Standards-Ethics/Standard-Definitions-(1).aspx. (accessed April 2019).

Bee, A. and J. Mitchell. 2017. “Do Older Americans Have More Income Than We Think?” SESHD Working Paper 2017, No. 39. Available at: https://www.census.gov/content/dam/Census/library/working-papers/2017/demo/SEHSD-WP2017-39.pdf. (accessed April 2019).

Board of Governors of the Federal Reserve System. 2017. Codebook for 2016 Survey of Consumer Finances. Available at: https://www.federalreserve.gov/econres/files/codebk2016.txt. (accessed April 2019).

Bollinger, C.R., B.T. Hirsch, C.M. Hokayem, and J.P. Ziliak. 2015. “Trouble in the Tails? What We Know about Earnings Nonresponse Thirty Years after Lillard, Smith, and Welch.” University of Kentucky Center for Poverty Research Discussion Paper Series, no. 120. Available at: https://uknowledge.uky.edu/ukcpr_papers/120/. (accessed April 2019).

Bound, J. and A.B. Krueger. 1991. “The Extent of Measurement Error in Longitudinal Earnings Data: Do Two Wrongs Make a Right?” Journal of Labor Economics 9: 1–24. Doi: http://dx.doi.org/10.1086/298256.

Bricker, J., L.J. Dettling, A. Henriques, J.W. Hsu, L. Jacobs, K.B. Moore, S. Pack, J. Sabelhaus, J. Thompson, and R.A. Windle. 2017. “Changes in U.S. Family Finances from 2013 to 2016: Evidence from the Survey of Consumer Finances.” Federal Reserve Bulletin 100(4). Available at: https://www.federalreserve.gov/publications/2017-September-changes-in-us-family-finances-from-2013-to-2016.htm. (accessed April 2019).

Czajka, J.L. and A. Beyler. 2016. Background Paper Declining Response Rates in Federal Surveys: Trends and Implications. Washington: Mathematica Policy Research. Available at: https://aspe.hhs.gov/system/files/pdf/255531/Decliningresponserates.pdf. (accessed April 2019).

Eggleston, J. and M. Gideon. 2017. “Evaluating Wealth Data Quality in the Redesigned 2014 Panel of the Survey of Income and Program Participation.” SIPP Working Paper, no. 278. Available at: https://www.census.gov/library/working-papers/2017/demo/SEHSD-WP2017-35.html. (accessed April 2019).

Ferber, R. 1966. “Item Nonresponse in a Consumer Survey.” Public Opinion Quarterly 30(3): 399–415. Doi: https://doi.org/10.1086/267432.

Gottschalck, A.O. and J.C. Moore. 2007. “Evaluation of Questionnaire Design Changes on Life Insurance Policy Data.” U.S. Census Research Report Series, Survey Methodology 2007(14). Available at: https://census.gov/library/working-papers/2007/adrm/rsm2007-14.html. (accessed April 2019).

Groves, R.M. and E. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias a Meta-Analysis.” Public Opinion Quarterly 72(2): 167–189. Doi: https://doi.org/10.1093/poq/nfn011.

Hokayem, C., T. Raghunathan, and J. Rothbaum. 2017. “Ignorable Nonresponse? Improved Imputation and Administrative Data in the CPS ASEC.” Paper presented at the Association for Public Policy Analysis and Management Fall Research Conference, November 2–4, 2017. Available at: https://appam.confex.com/appam/2017/webprogram/Paper23281.html. (accessed April 2019).

Hsu, J.W. M.D. Schmeiser, C. Haggerty, and S. Nelson. 2017. “The Effect of Large Monetary Incentives on Survey Completion: Evidence from a Randomized Experiment with the Survey of Consumer Finances.” Public Opinion Quarterly 81(3): 736–747. Doi: https://doi.org/10.1093/poq/nfx006.

Lillard, Lee, James P. Smith, and Finis Welch. 1986. “What do we really know about wages? The importance of nonreporting and census imputation.” Journal of Political Economy 94, no. 3, Part 1: 489–506. Doi: https://doi.org/10.1086/261386.

National Research Council. 2009. Reengineering the Survey of Income and Program Participation. Constance F. Citro and John Karl Scholz, eds. Committee on National Statistics, Division of Behavioral and Social Sciences Education. Washington, DC: The National Academies Press.

U.S. Census Bureau. 2016. Survey of Income and Program Participation 2014 Panel Users’ Guide. Washington, D.C., U.S.A. Available at: https://www.census.gov/programssurveys/sipp/guidance/users-guide.html. (accessed April 2019).

U.S. Census Bureau. 2017. Wealth, Asset Ownership, & Debt of Households Detailed Tables: 2013. Washington, D.C. U.S.A. Available at: https://www.census.gov/topics/income-poverty/wealth/data/tables.html. (accessed April 2019).

Watson, D., L.A. Clark, and A. Tellegen. 1998. “Development and Validation of Brief Measures of Positive and Negative Affect: the PANAS scales.” Journal of Personality and Social Psychology 54(6): 1063–1070. Doi: http://doi.apa.org/journals/psp/54/6/1063.html.

Westra, A., M. Sundukchi, and T. Mattingly. 2015. “Designing a Multipurpose Longitudinal Incentives Experiment for the Survey of Income and Program Participation.” In Proceedings of the 2015 Federal Committee on Statistical Methodology (FCSM) Research Conference, December 1–3, 2015. Washington, DC: Federal Committee on Statistical Methodology. Available at: https://s3.amazonaws.com/sitesusa/wp-content/uploads/sites/242/2016/03/E3_Westra_2015FCSM.pdf. (accessed April 2019).

Yan, T., R. Curtin, and M. Jans. 2010. “Trends in Income Nonresponse Over Two Decades.” Journal of Official Statistics 26(1): 145–164. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/trends-in-income-nonresponseover-two-decades.pdf. (accessed April 2019).

Journal of Official Statistics

The Journal of Statistics Sweden

Journal Information


IMPACT FACTOR 2018: 0,837
5-year IMPACT FACTOR: 0,934

CiteScore 2018: 1.04

SCImago Journal Rank (SJR) 2018: 0.963
Source Normalized Impact per Paper (SNIP) 2018: 1.020

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 149 149 119
PDF Downloads 123 123 105