This article describes a general framework for improving the quality of statistical programs in organizations that provide a continual flow of statistical products to users and stakeholders. The work stems from a 2011 mandate to Statistics Sweden issued by the Swedish Ministry of Finance to develop a system of quality indicators for tracking developments and changes in product quality and for achieving continual improvements in survey quality across a diverse set of key statistical products. We describe this system, apply it to a number of products at Statistics Sweden, and summarize key results and lessons learned. The implications of this work for monitoring and evaluating product quality in other statistical organizations are also discussed.
Andersen, R., J. Kaspar, and M. Frankel. 1979. Total Survey Error. San Francisco: Jossey-Bass Publishers.
Baldrige Performance Excellence Program 2013. The 2013-2014 Criteria for Performance Excellence. Available at: http://www.nist.gov/baldrige/ (accessed August 3, 2013).
Barkley, B.T. 2004. Project Risk Management. New York: McGraw Hill Professional.
Biemer, P. 2011. Latent Class Analysis of Survey Error. Hoboken, NJ: John Wiley & Sons.
Biemer, P. 2014. “Comment on ‘On Information Quality’ by Kenett and Shmueli.” Journal of the Royal Statistical Society, Series A. Vol. 177, Part 1: 27-29.
Biemer, P. and L. Lyberg. 2003. Introduction to Survey Quality. New York: John Wiley & Sons.
Biemer, P. and D. Trewin. 2012. Development of Quality Indicators at Statistics Sweden. Report to Statistics Sweden, January 2012.
Biemer, P. and D. Trewin. 2013. A Second Application of the ASPIRE Quality Evaluation System for Statistics Sweden. Report to Statistics Sweden, January 2013.
Biemer, P. and D. Trewin. 2014. A Third Application of ASPIRE for Statistics Sweden. Report to Statistics Sweden, January 2014.
Brackstone, G. 1999. “Managing Data Quality in a Statistical Agency.” Survey Methodology 25: 139-149.
Breyfogle, F. 2003. Implementing Six Sigma, 2nd edition. New York: John Wiley & Sons.
Conley-Tyler, M. 2005. “A Fundamental Choice: Internal or External Evaluation?” Evaluation Journal of Australasia 4: 3-11.
COSO, 2004. Enterprise Risk Management - Integrated Framework. Available at: http:// www.coso.org/documents/coso_erm_executivesummary.pdf (accessed August 3, 2013).
COSO, 2013. Internal Control - Integrated Framework, 2013. Available at: http://www.coso.org/documents/coso%202013%20icfr%20executive_summary.pdf (accessed August 3, 2013).
Couper, M. and L. Lyberg. 2005. “The Use of Paradata in Survey Research.” In Proceedings of the 55th Session of the International Statistical Institute, Sydney, Australia, April 7, 2005. Available at: http://isi.cbs.nl/iamamember/CD6-Sydney2005/ISI_Final_Proceedings.htm (accessed June 26, 2014).
Curtin, R., S. Presser, and E. Singer. 2000. “The Effects of Response Rate Changes on the Index of Consumer Sentiment.” Public Opinion Quarterly 64: 413-428.
Dalenius, T. 1967. Nonsampling Errors in Census and Sample Surveys. Report no. 5 in the research project Errors in Surveys, Stockholm University.
Deming, E. 1944. “On Errors in Surveys.” American Sociological Review 9: 359-369.
Deming, E. 1986. Out of the Crisis. Cambridge, MA: MIT Press.
EFQM, 2013. “An Overview of the Excellence Model.” Available at: https://www.google.com/url?q=http://www2.efqm.org/en/PdfResources/EFQM%2520Excellence%2520Model%25202013%520EN%2520extract.pdf&sa=U&ei=9BasU4nkHsqTqAbUhIGQCg&ved=0CAUQFjAA&client=internal-uds-cse&usg=AFQjCNHthJnhRPIS1t6cfa4Ka9ePXOLRxg (accessed June 26, 2014).
Eltinge, J., P. Biemer, and A. Holmberg. 2013. “A Potential Framework for Integration of Architecture and Methodology to Improve Statistical Production Systems.” Journal of Official Statistics 29: 125-145. DOI: http://dx.doi.org/10.2478/jos-2013-0007.
European Statistical System (ESS) 2011. “Quality Assurance Framework of the European Statistical System, Version 1.1.” Available at: http://epp.eurostat.ec.europa.eu/cache/ITY_PUBLIC/QAF_2012/EN/QAF_2012-EN.PDF (accessed August 9, 2013).
Eurostat 2005. “European Statistics Code of Practice, Revised Edition.” Available at: http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KS-32-11-955/EN/KS-32-11-955-EN.PDF (accessed June 26, 2014).
Eurostat 2009. Regulation (EC) No 223/2009 of the European Parliament and of the Council of 11 March 2009, Eurostat General/Standard report, Luxembourg, April 4-5. Available at: http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:R0223 (accessed June 18, 2014).
Gonzales, M.E., J.L. Ogus, G. Shapiro, and B.J. Tepping. 1975. “Standards for Discussion and Presentation of Errors in Surveys and Census Data.” Journal of American Statistical Association 70: 5-23.
Groves, R.M. and L.E. Lyberg. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly 74: 849-879. DOI:http://dx.doi.org/10.1093/poq/nfq065.
Hansen, M., W. Hurwitz, and W. Madow. 1953. Sample Survey Methods and Theory, Volumes I and II. New York: John Wiley & Sons.
Hansen, M., W. Hurwitz, and L. Pritzker. 1967. Standardization of Procedures for the Evaluation of Data: Measurement Errors and Statistical Standards in the Bureau of the Census. Paper presented at the 36th session of the International Statistical Institute.
Imai, M. 1986. Kaisen: the Key to Japan’s Competitive Success. New York: McGraw-Hill Education.
International Monetary Fund (IMF) 2003. Data Quality Assessment Framework and Data Quality Program. Available at: http://www.imf.org/external/np/sta/dsbb/2003/eng/dqaf.htm (accessed June 21, 2013).
International Standards Organization 2006. Market, Opinion and Social Research ISO Standard No. 20252. Available at: www.standards.org/standards/listing/iso_20252 (accessed August 8, 2014).
International Standards Organization 2009. Risk Mangement: Principles and Guidelines for Implementation, ISO/DIS 31000 Standard No. 31000. Available at: www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=43170 (accessed August 8, 2014).
Journal of Official Statistics 2013. Special Issue on Systems and Architectures for High-Quality Statistics Production, edited by B. Lorenc, I. Jansson, P. Biemer, J. Eltinge, and A. Holmberg, Vol. 1, March, 2013.
Juran, J. and B. Godfrey. 1999. Juran’s Quality Handbook. New York: McGraw-Hill.
Karsak, E.E. 2004. “Fuzzy Multiple Objective Decision Making Approach to Prioritize Design Requirements in Quality Function Deployment.” International Journal of Production Research 42: 3957-3974.
Keeter, S., C. Miller, A. Kohut, R. Groves, and S. Presser. 2000. “Consequences of Reducing Nonresponse in a Large National Telephone Survey.” Public Opinion Quarterly 64: 125-148. DOI: http://dx.doi.org/10.1086/317759.
Kenett, R.S. and G. Shmueli. 2014. “On Information Quality.” Journal of the Royal Statistical Society, Series A 177: 3-38. DOI:http://dx.doi.org/10.1111/rssa.12007.
Kish, L. 1962. “Studies of Interviewer Variance for Attitudinal Variables.” Journal of the American Statistical Association 57: 92-115. Lequiller, F and D. Blades. 2006. Understanding National Accounts. Paris: OECD 2006. Available at: http://www.eastafritac.org/images/uploads/documents_storage/Understanding_National_Accounts_-_OECD.pdf (accessed June 21, 2013).
Lyberg, L. and P. Biemer. 2008. “Quality Assurance and Quality Control in Surveys.” In International Handbook on Survey Methodology, edited by J. Hox, E. de Leeuw, and D. Dillman, 421-441. Mahwah, NJ: Lawrence Erlbaum Associates.
Lyberg, L., L. Japec, and P. Biemer. 1998. “Quality Improvement in Surveys - A Process Perspective.” In Proceedings of the Survey Research Methods Section of the American Statistical Association, 23-31.
Lyberg, L. 2012. “Survey Quality.” Survey Methodology 38: 107-130.
McDavid, J., I. Huse, and L. Hawthorn. 2013. Program Evaluation and Performance Measurement: An Introduction to Practice, Second Edition. New York: Sage Publications.
Michalek, J.J., O. Ceryan, P.Y. Papalambros, and Y. Koren. 2006. “Balancing Marketing and Manufacturing Objectives in Product Line Design.” ASME Journal of Mechanical Design 128: 1196-1204. DOI: http://dx.doi.org/10.1115/1.2336252.
Merkle, D. and M. Edelman. 2002. “Nonresponse in Exit Polls: A Comprehensive Analysis.” In Survey Nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R. Little, 243-257. New York: John Wiley and Sons.
Morganstein, D. and D. Marker. 1997. “Continuous Quality Improvement in Statistical Agencies.” In Survey Measurement and Process Quality, edited by L. Lyberg, P. Biemer, M. Collins, E. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin, 475-500. New York: Wiley and Sons.
Nealon, J. and E. Gleaton. 2013. “Consolidation and Standardization of Survey Operations at a Decentralized Federal Statistical Agency.” Journal of Official Statistic 29: 5-28. DOI: http://dx.doi.org/10.2478/jos-2013-0002.
Neyman, J. 1934. “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection.” Journal of the Royal Statistical Society 97: 558-606.
Neyman, J. 1938. Lectures and Conferences on Mathematical Statistics and Probability. Washington, DC: U.S. Department of Agriculture.
Organisation for Economic Cooperation and Development (OECD) 2011. Quality Framework and Guidelines for OECD Statistical Activities. Available at: http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=std/qfs%282011%291&doclanguage= en (accessed June 21, 2013).
Office of National Statistics (ONS) 2007. Guidelines for Measuring Statistical Quality, Version 3.1. Available at: http://www.ons.gov.uk/ons/guide-method/method-quality/quality/guidelines-for-measuring-statistical-quality/index.html (accessed June 21, 2013).
Rossi, P.H., W.M. Lipsey, and H.E. Freeman. 2004. Evaluation: A Systematic Approach. 7th ed. Thousand Oaks, CA: Sage Publishers.
Seyb, A., R. McKenzie, and A. Skerrett. 2013. “Innovative Production Systems at New Zealand: Overcoming the Design and Build Bottleneck.” Journal of Official Statistics 29: 73-97. DOI: http://dx.doi.org/10.2478/jos-2013-0005.
Statistics Canada 2009. Statistics Canada Quality Guidelines, Fifth Edition. Available at: http://www5.statcan.gc.ca/bsolc/olc-cel/olc-cel?catno=12-539-X&CHROPG=1&lang=eng (accessed March 10, 2014).
Statistiska centralbyra°n 2001. Quality Definition and Recommendations for Quality Declarations of Official Statistics. Available at: http://www.scb.se/Grupp/Hitta_statistik/Forsta_Statistik/Metod/_Dokument/MIS2001_1.pdf (accessed June 18, 2014).
Stephan, F.F. 1948. “History of the Uses of Modern Sampling Procedures.” Journal of the American Statistical Association 43: 12-39.
Struijs, P., A. Camstra, R. Renssen, and B. Braaksma. 2013. “Redesign of Statistics Production within an Architectural Framework: The Dutch Experience.” Journal of Official Statistics 29: 49-71. DOI: http://dx.doi.org/10.2478/jos-2013-0004.
U.S. Bureau of the Census 1974. “Technical Paper 32: Standards for Discussion and Presentation of Errors in Data. U.S. Department of Commerce.” U.S. Government Printing Office, Technical Paper 32, Department of Commerce.
U.S. Office of Management and Budget 2002. “Guidelines for Ensuring, and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies.” Federal Register, 67, 36, February 22.