This article discusses the critical and complex design decisions associated with transitioning an interviewer-administered survey to a self-administered, postal, web/paper survey. Our approach embeds adaptive, responsive, and tailored (ART) design principles and data visualization during a multi-phased data collection operation to project the outcomes of each phase in preparation for subsequent phases. This requires rapid decision making based upon experimental results using a data visualization system to monitor critical-to-quality (CTQ) metrics and facilitate projections of outcomes from the current phase of data collection to inform the design of the subsequent phase. We describe the objectives of the overall design, the features designed to address these objectives, components of the visual adaptive total design (ATD) system for monitoring quality components and relative costs in real time, and examples of the visualization elements and functionalities that were used in one case study. We also discuss subsequent initiatives to develop an interactive version of the monitoring tool and applications for other studies, including those employing adaptive, responsive, and tailored (ART) designs. Our case study is a series of pilot studies conducted for the Residential Energy Consumption Survey (RECS), sponsored by the U.S. Energy Information Administration (EIA).
If the inline PDF is not rendering correctly, you can download the PDF file here.
AAPOR (The American Association for Public Opinion Research). 2015. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys 8th edition. Oakbrook Terrace IL: AAPOR.
Amaya A. P. Biemer and D. Kinyon. 2017. “Total Error in a Big Data World with Applications to the Residential Energy Consumption Survey.” Presented at the American Association for Public Opinion Research Annual Conference New Orleans LA.
Biemer P. 2010. “Total Survey Error: Design Implementation and Evaluation.” Public Opinion Quarterly 74(5): 817–848. Doi: https://doi.org/10.1093/poq/nfq058.
Biemer P. K.H. Harris B. Burke K. Considine C. Halpern and C. Suchindran. 2017a. “Transitioning an In-Person Longitudinal Survey to a Mixed-Mode Two-Phase Survey Design: Preliminary Results.” Presented at the Annual Conference of the American Association for Public Opinion Research. New Orleans LA.
Biemer P. J. Murphy S. Zimmer C. Berry G. Deng and K. Lewis. 2017b. “Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys.” Journal of Survey Statistics and Methodology. Doi: https://doi.org/10.1093/jssam/smx015.
Breyfogle F. 2003. Implementing Six Sigma: Smarter Solutions Using Statistical Methods. Hoboken NJ: John Wiley & Sons.
Camoes J. 2008. How to Create a Thematic Map in Excel. Available at: http://www.excelcharts.com/blog/how-to-create-thematic-map-excel/ (accessed November 26 2017).
Chun A.Y. B. Schouten and J. Wagner. 2017. “JOS Special Issue on Responsive and Adaptive Survey Design: Looking Back to See Forward – Editorial.” Journal of Official Statistics 33(3): 571–577. Doi: http://dx.doi.org/10.1515/JOS-2017-0027.
Cleveland W. 1993. Visualizing Data. Summit NJ: Hobart Press.
Cramér H. 1946. Mathematical Methods of Statistics. Princeton: Princeton University Press.
Dillman D. J.D. Smyth and L.M. Christian. 2014. Internet Phone and Mail and Mixed-Mode Surveys: The Tailored Design Method 4th Edition. Hoboken NJ: Wiley.
Dillman D.A. and M.L. Edwards. 2016. “Designing a Mixed-Mode Survey.” In Wolfe Christof Joye Dominique Smith Tom W. and Fu Yang-chih Sage Handbook of Survey Methodology. Sage Publications Wolf Joye Smith and Fu. Thousand Oaks. CA 255–268.
Duprey M. J. Murphy P. Biemer and R. Chew. 2017. “Veni Vidi Vici: Interactive Data Visualizations for Adaptive Total Design.” Presented at the 5th Workshop on Adaptive and Responsive Survey Design. Ann Arbor MI.
Eddy W.F. and Marton K. Editors. 2012. Effective Tracking of Building Energy Use: Improving the Commercial Buildings and Residential Energy Consumption Surveys. Washington D.C.: The National Academies Press.
Edgar J. J. Murphy and M. Keating. 2016. “Comparing Traditional and Crowdsourcing Methods for Pretesting Survey Questions.” SAGE Open 6(4): 1–14. Doi: https://doi.org/10.1177/2158244016671770.
Groves R.M. 1989. Survey Errors and Survey Costs. New York: Wiley.
Groves R. and S. Heeringa. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society Series A 169(3): 439–457. Doi: http://dx.doi.org/10.1111/j.1467-985X.2006.00423.x.
Hardin M. D. Horn R. Perez and L. Williams. 2012. “Which Chart or Graph is Right for You? Telling Impactful Stories with Data.” Tableau Software. Available at: http://theathenaforum.org/sites/default/files/WHich%20chart%20is%20right%20for%20you.pdf (accessed November 26 2017).
Hornbaek K. and E. Frokjaer. 2003. “Reading Patterns and Usability in Visualizations of Electronic Documents.” ACM Transactions on Computer-Human Interaction 10(2): 119–149. Doi: https://doi.org/10.1145/772047.772050.
Howden L. S. Joestl and R. Cohen. 2015. Improving Response Rates using a Mixed-Mode Approach: Results from the National Health Care Interview Survey. Presented at the 2015 FedCASIC Conference. Available at: https://www.census.gov/fedcasic/fc2015/ppt/27_howden.pdf (accessed November 21 2017).
Kahneman D. 2011. Thinking Fast and Slow. New York: Farrar Straus and Giroux.
Laflamme F. and J. Wagner. 2016. “Responsive and Adaptive Designs.” In The SAGE Handbook of Survey Methodology edited by C. Wolf D. Joye T. Smith and Y. Fu. Los Angeles: Sage.
Link M. and A. Mokdad. 2005. “Alternative Modes for Health Surveillance Surveys: an Experiment with Web Mail and Telephone.” Epidemiology 16: 701–704. Doi: 10.1097/01.ede.0000172138.67080.7f.
Luiten A. and B. Schouten. 2013. “Tailored Fieldwork Design to Increase Representative Household Survey Response: an Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society A 176: 169–189. Doi: https://doi.org/10.1111/j.1467-985X.2012.01080.x.
Morganstein D.R. and D.A. Marker. 1997. “Continuous Quality Improvement in Statistical Agencies.” In Survey Measurement and Process Quality edited by L.E. Lyberg P. Biemer M. Collins E.D. de Leeuw C. Dippo N. Schwarz and D. Trewin. (pp. 475–500). New York: John Wiley & Sons.
Murphy J. D. Mayclin A. Richards and D. Roe. 2016. “A Multi-method Approach to Survey Pretesting.” In 2015 FCSM Research Conference Proceedings. Available at: https://fcsm.sites.usa.gov/files/2016/03/D3_Murphy_2015FCSM.pdf. (accessed November 26 2017).
Murphy J. P. Biemer M. Duprey and R. Chew. 2017. “Interactive Adaptive Total Design Reports for Near Real-Time Survey Monitoring.” Presented at the 2017 Conference of the European Survey Research Association. Lisbon Portugal.
Schouten B. F. Cobben and J. Bethlehem. 2009. “Indictators of Representativeness of Survey Nonresponse.” Survey Methodology 35: 101–113.
Schouten B. A. Peytchev and J. Wagner. 2017. Adaptive Survey Design. Boca Raton FL: Chapman and Hall/CRC.
Tufte E. 2001. The Visual Display of Quantitative Information (2nd ed.). Cheshire CT: Graphics Press. ISBN 0-9613921-4-2.
U.S. Census Bureau. 2015. American Community Survey (ACS) 2014 Data Release New and Noteable. Available at: https://www.census.gov/programs-surveys/acs/news/data-releases/2014/release.html#par_textimage_12. (accessed November 21 2017).
Zimmer S. P. Biemer P. Kott and C. Berry. 2016. “Testing a Model-Directed Mixed Mode Protocol in the RECS Pilot Study.” In 2015 FCSM Research Conference Proceedings. Available at: https://s3.amazonaws.com/sitesusa/wp-content/uploads/sites/242/2016/03/G2_Zimmer_2015FCSM.pdf. (accessed November 26 2017).