Measuring Trust in Medical Researchers: Adding Insights from Cognitive Interviews to Examine Agree-Disagree and Construct-Specific Survey Questions

Anderson, L.A. and R.F. Dedrick. 1990. “Development of the Trust in Physician Scale: A Measure to Assess Interpersonal Trust in Patient-physician Relationships.” Psychological Reports 67: 1091–1100. Doi: https://doi.org/10.2466/pr0.1990.67.3f.1091.10.2466/pr0.1990.67.3f.1091Open DOISearch in Google Scholar

Audacity Developer Team. 2008. Audacity (Version 1.2.6) [Computer Software]: Available at: http://www.audacityteam.org/download/ (accessed April 2019).Search in Google Scholar

Bassili, J.N. and B.S. Scott. 1996. “Response Latency as a Signal to Question Problems in Survey Research.” Public Opinion Quarterly 60: 390–399. Doi: https://doi.org/10.1086/297760.10.1086/297760Open DOISearch in Google Scholar

Braunstein, J.B., N.S. Sherber, S.P. Schulman, E.L. Ding, and N.R. Powe. 2008. “Race, Medical Researcher Distrust, Perceived Harm, and Willingness to Participate in Cardiovascular Prevention Trials.” Medicine 87: 1–9. Doi: https://doi.org/10.1097/MD.0b013e3181625d78.10.1097/MD.0b013e3181625d78Open DOISearch in Google Scholar

Carpenter, P.A. and M.A. Just. 1975. “Sentence Comprehension: A Psycholinguistic Processing Model of Verification.” Psychological Review 82: 45–73. Available at: http://psycnet.apa.org/doi/10.1037/h0076248 (accessed April 2019).10.1037/h0076248(2019)Open DOISearch in Google Scholar

Corbie-Smith, G., S.B. Thomas, and D.M.M. St. George. 2002. “Distrust, Race, and Research.” Archives of Internal Medicine 162: 2458–2463. Doi: https://doi.org/10.1001/archinte.162.21.2458.10.1001/archinte.162.21.2458Open DOISearch in Google Scholar

Davidov, E., B. Meuleman, J. Cieciuch, P. Schmidt, and J. Billiet. 2014. “Measurement Equivalence in Cross-National Research.” Annual Review of Sociology 40: 55–75. Doi: https://doi.org/10.1146/annurev-soc-071913-043137.10.1146/annurev-soc-071913-043137Open DOISearch in Google Scholar

De Leeuw, E. and N. Berzelak. 2016. “Survey Mode or Survey Modes?” In The SAGE Handbook of Survey Methodology, edited by C. Wolf, J. Dominique, T.W. Smith, and F. Yang-chih, 142–156. Los Angeles: SAGE Publications Ltd. Available at: https://books.google.com/books?hl=en&lr=&id=g8OMDAAAQBAJ&oi=fnd&pg=PA142&dq=survey+mode+or+modes+berzelak&ots=DyqMiBT1oS&sig=hGg7pa80-bI535N5GgSUwvLmLfY#v=onepage&q=survey%20mode%20or%20modes%20berzelak&f=false (accessed April 2019).Search in Google Scholar

Dijkstra, W. and Y. Ongena. 2006. “Question-Answer Sequences in Survey-Interviews.” Quality & Quantity 40: 983–1011. Doi: https://doi.org/10.1007/s11135-005-5076-4.10.1007/s11135-005-5076-4Open DOISearch in Google Scholar

Dillman, D.A., J.D. Smyth, and L.M. Christian. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (4th edition). Hoboken, NJ: John Wiley.Search in Google Scholar

Draisma, S. and W. Dijkstra. 2004. “Response Latency and (Para)linguistic Expression as Indicators of Response Error.” In Methods for Testing and Evaluating Survey Questionnaires, edited by S. Presser, J.M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin, J. Martin, and E. Singer, 131–148. New York: Springer-Verlag. Doi: https://doi.org/10.1002/0471654728.ch7.10.1002/0471654728.ch7Open DOISearch in Google Scholar

Dykema, J., J.M. Lepkowski, and S. Blixt. 1997. “The Effect of Interviewer and Respondent Behavior on Data Quality: Analysis of Interaction Coding in a Validation Study.” In Survey Measurement and Process Quality, edited by L. Lyberg, P. Biemer, M. Collins, E. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin, 287–310. N.Y: Wiley-Interscience. Available at: https://onlinelibrary.wiley.com/doi/10.1002/9781118490013.ch12 (accessed April 2019).10.1002/9781118490013.ch12(accessed2019)Open DOISearch in Google Scholar

Dykema, J., N.C. Schaeffer, and D. Garbarski. 2012. “Effects of Agree-Disagree Versus Construct-Specific Items on Reliability, Validity, and Interviewer-Respondent Interaction.” Presented at the American Association for Public Opinion Research, May 17–20. 2012. Orlando, Florida, U.S.A.Search in Google Scholar

Dykema, J., N.C. Schaeffer, and D. Garbarski. 2019. “Towards a Reconsideration of the Use of Agree-Disagree Questions in Measuring Subjective Evaluations.” Unpublished manuscript, University of Wisconsin-Madison, Madison-WI.Search in Google Scholar

Edwards, D.F. 2015. “Voices Heard.” Presented at the Health Equity Leadership Institute, Madison, WI.Search in Google Scholar

Egede, L.E. and C. Ellis. 2008. “Development and Testing of the Multidimensional Trust in Health Care Systems Scale.” Journal of General Internal Medicine 23: 808–815. Doi: https://doi.org/10.1007/s11606-008-0613-1.10.1007/s11606-008-0613-1251787218415653Open DOISearch in Google Scholar

Fleiss, J.L. 1981. Statistical Methods for Rates and Proportions, 2nd edition. New York: Wiley.Search in Google Scholar

Fortune-Greeley, A.K., K.E. Flynn, D.D. Jeffery, M.S. Williams, F.J. Keefe, R.B. Reeve, G.B. Willis, and K.P. Weinfurt. 2009. “Using Cognitive Interviews to Evaluate Items for Measuring Sexual Functioning Across Cancer Populations: Improvements and Remaining Challenges.” Quality of Life Research 18: 1085–1093. Doi: https://doi.org/10.1007/s11136-009-9523-x.10.1007/s11136-009-9523-xSearch in Google Scholar

Fowler, F.J. and C. Cosenza. 2009. “Design and Evaluation of Survey Questions.” In The Sage Handbook of Applied Social Research Methods, edited by L. Bickman and D.J. Rog, 375–412. Thousand Oaks, CA: Sage.10.4135/9781483348858.n12Search in Google Scholar

Hall, M.A., F. Camacho, E. Dugan, and R. Balkrishnan. 2002a. “Trust in the Medical Profession: Conceptual and Measurement Issues.” Health Services Research 37: 1419–1439. Doi: https://doi.org/10.1111/1475-6773.01070.10.1111/1475-6773.01070Open DOISearch in Google Scholar

Hall, M.A., F. Camacho, J.S. Lawlor, V. DePuy, J. Sugarman, and K. Weinfurt. 2006. “Measuring Trust in Medical Researchers.” Medical Care 44: 1048–1053. Available at: http://www.jstor.org/stable/41219560 (accessed April 2019).10.1097/01.mlr.0000228023.37087.cbSearch in Google Scholar

Hall, M.A., E. Dugan, B. Zheng, and A.K. Mishra. 2001. “Trust in Physicians and Medical Institutions: What is It, Can It be Measured, and Does It Matter?” Milbank Quarterly 79: 613–639. Doi: https://doi.org/10.1111/1468-0009.00223.10.1111/1468-0009.00223Open DOISearch in Google Scholar

Hall, M.A., B. Zheng, E. Dugan, F. Camacho, K.E. Kidd, A. Mishra, and R. Balkrishnan. 2002b. “Measuring Patients’ Trust in their Primary Care Providers.” Medical Care Research and Review 59: 293–318.Doi: https://doi.org/10.1177/1077558702059003004.10.1177/1077558702059003004Open DOISearch in Google Scholar

Hanson, T. 2015. “Comparing Agreement and Item-Specific Response Scales: Results from an Experiment.” Social Research Practice 1: 17–25. Available at: http://the-sra.org.uk/wp-content/uploads/social-research-practice-journal-issue-01-winter-2015.pdf (accessed April 2019).Search in Google Scholar

Hayman, R.M., B.J. Taylor, N.S. Peart, B.C. Galland, and R.M. Sayers. 2001. “Participation in Research: Informed Consent, Motivation and Influence.” Journal of Paediatrics and Child Health 37: 51–54. Available at: https://doi.org/10.1046/j.1440-1754.2001.00612.x (accessed April 2019).10.1046/j.1440-1754.2001.00612.x(accessed2019)Open DOISearch in Google Scholar

Henderson, G., J. Garrett, J. Bussey-Jones, M.E. Moloney, C. Blumenthal, and G. Corbie-Smith. 2008. “Great Expectations: Views of Genetic Research Participants Regarding Current and Future Genetic Studies.” Genetics in Medicine 10: 193–200. Doi: https://doi.org/10.1097/GIM.0b013e318164e4f5.10.1097/GIM.0b013e318164e4f518344709Search in Google Scholar

Höhne, J.K. and D. Krebs. 2018. “Scale Direction Effects in Agree/Disagree and Item-Specific Questions: A Comparison of Question Formats.” International Journal of Social Research Methodology 21: 91–103. Doi: https://doi.org/10.1080/13645579.2017.1325566.10.1080/13645579.2017.1325566Open DOISearch in Google Scholar

Höhne, J.K. and T. Lenzner. 2018. “New Insights on the Cognitive Processing of Agree/Disagree and Item-Specific Questions.” Journal of Survey Statistics and Methodology 6: 401–417. Doi: https://doi.org/10.1093/jssam/smx028.10.1093/jssam/smx028Open DOISearch in Google Scholar

Höhne, J.K., S. Schlosser, and D. Krebs. 2017. “Investigating Cognitive Effort and Response Quality of Question Formats in Web Surveys Using Paradata.” Field Methods 29: 365–382. Doi: https://doi.org/10.1177/1525822x17710640.10.1177/1525822x17710640Open DOISearch in Google Scholar

Holbrook, A.L. 2008. “Recency Effect.” In Encyclopedia of Survey Research Methodology, edited by P.J. Lavrakas, 695–696. Newbury Park, CA: Sage.Search in Google Scholar

Johnson, R.B. and A.J. Onwuegbuzie. 2004. “Mixed Methods Research: A Research Paradigm Whose Time Has Come.” Educational Researcher 33: 14–26. Doi: https://doi.org/10.3102/0013189X033007014.10.3102/0013189X033007014Open DOISearch in Google Scholar

Krosnick, J.A. and S. Presser. 2010. “Question and Questionnaire Design.” In Handbook of Survey Research, Second Edition, edited by P.V. Marsden and J.D. Wright, 263–313. Bingley, UK: Emerald Group Publishing Limited.Search in Google Scholar

Kuru, O. and J. Pasek. 2016. “Improving Social Media Measurement in Surveys: Avoiding Acquiescence Bias in Facebook Research.” Computers in Human Behavior 57: 82–92. Available at: https://doi.org/10.1016/j.chb.2015.12.008 (accessed April 2019).10.1016/j.chb.2015.12.008(accessed2019)Open DOISearch in Google Scholar

Landis, J.R. and G.G. Koch. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics 33: 159–174. Doi: https://doi.org/10.2307/2529310.10.2307/2529310Open DOISearch in Google Scholar

Lelkes, Y. and R. Weiss. 2015. “Much Ado about Acquiescence: The Relative Validity and Reliability of Construct-Specific and Agree-Disagree Questions.” Research and Politics 2: 1–8. Doi: https://doi.org/10.1177/2053168015604173.10.1177/2053168015604173Open DOISearch in Google Scholar

Liu, M., S. Lee, and F.G. Conrad. 2015. “Comparing Extreme Response Styles between Agree-Disagree and Item-Specific Scales.” Public Opinion Quarterly 79: 952–975. Doi: https://doi.org/10.1093/poq/nfv034.10.1093/poq/nfv034Open DOISearch in Google Scholar

Mainous, A.G., D.W. Smith, M.E. Geesey, and B.C. Tilley. 2006. “Development of a Measure to Assess Patient Trust in Medical Researchers.” Annals of Family Medicine 4: 247–252. Doi: https://doi.org/10.1370/afm.541.10.1370/afm.541147944516735527Open DOISearch in Google Scholar

Revilla, M. and C. Ochoa. 2015. “Quality of Different Scales in an Online Survey in Mexico and Columbia.” Journal of Politics in Latin America 7: 157–177. Available at: https://journals.sub.uni-hamburg.de/giga/jpla/article/view/903/910 (accessed April 2019).10.1177/1866802X1500700305Search in Google Scholar

Rogers, W. 1994. “Regression Standard Errors in Clustered Samples.” Stata Technical Bulletin 13. Available at: https://ideas.repec.org/a/tsj/stbull/y1994v3i13sg17.html (accessed April 2019).Search in Google Scholar

Ryan, G.W. and H.R. Bernard. 2003. “Techniques to Identify Themes.” Field Methods 15: 85–109. Doi: https://doi.org/10.1177/1525822x02239569.10.1177/1525822x02239569Open DOISearch in Google Scholar

Saris, W.E., M. Revilla, J.A. Krosnick, and E.M. Shaeffer. 2010. “Comparing Questions with Agree/Disagree Response Options to Questions with Item-Specific Response Options.” Survey Research Methods 4: 61–79. Doi: https://ojs.ub.uni-konstanz.de/srm/article/view/2682/3971.Search in Google Scholar

Schaeffer, N.C. and J. Dykema. 2011. “Response 1 to Fowler’s Chapter: Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions.” In Question Evaluation Methods: Contributing to the Science of Data Quality, edited by J. Madans, K. Miller, A. Maitland, and G. Willis, 23–39. Hoboken, NJ: John Wiley & Sons, Inc. Available at: https://doi.org/10.1002/9781118037003.ch3.10.1002/9781118037003.ch3Open DOISearch in Google Scholar

Scharff, D.P., K.J. Mathews, P. Jackson, J. Hoffsuemmer, E. Martin, and D. Edwards. 2010. “More than Tuskegee: Understanding Mistrust about Research Participation.” Journal of Health Care for the Poor and Underserved 21: 879–897. Doi: https://doi.org/10.1353/hpu.0.0323.10.1353/hpu.0.0323435480620693733Open DOISearch in Google Scholar

Schuman, H. and S. Presser. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Thousand Oaks, CA: Sage Publications, Inc.Search in Google Scholar

Smith, T.W., P.V. Marsden, and M. Hout. 2013. General Social Survey, 1972–2010 [Cumulative File]. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2013-02-07. Doi: https://doi.org/10.3886/ICPSR31521.v1.10.3886/ICPSR31521.v1Open DOISearch in Google Scholar

Streiner, D.L., G.R. Norman, and J. Cairney. 2015. Health Measurement Scales: A Practical Guide to Their Development and Use. Oxford, UK: Oxford University Press.10.1093/med/9780199685219.001.0001Search in Google Scholar

Sturgis, P., C. Roberts, and P. Smith. 2014. “Middle Alternatives Revisited: How the neither/nor Response Acts as a Way of Saying “I Don’t Know”?” Sociological Methods & Research 43: 15–38. Doi: https://doi.org/10.1177/0049124112452527.10.1177/0049124112452527Open DOISearch in Google Scholar

Thompson, H.S., H.B. Valdimarsdottir, G. Winkel, L. Jandorf, and W.W. Redd. 2004. “The Group-Based Medical Mistrust Scale: Psychometric Properties and Association with Breast Cancer Screening.” Preventive Medicine 38: 209–218. Doi: https://doi.org/10.1016/j.ypmed.2003.09.041.10.1016/j.ypmed.2003.09.04114715214Open DOISearch in Google Scholar

Tourangeau, R., M.C. Couper, and F. Conrad. 2004. “Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions.” Public Opinion Quarterly 68: 368–393. Doi: https://doi.org/10.1093/poq/nfh035.10.1093/poq/nfh035Open DOISearch in Google Scholar

Tourangeau, R., L.J. Rips, and K. Rasinski. 2000. The Psychology of Survey Response. Cambridge, England: Cambridge University Press.10.1017/CBO9780511819322Search in Google Scholar

Williams, M.M., D.P. Scharff, K.J. Mathews, J.S. Hoffsuemmer, P. Jackson, J.C. Morris, and D.F. Edwards. 2010. “Barriers and Facilitators of African American Participation in Alzheimer Disease Biomarker Research.” Alzheimer Disease & Associated Disorders 24: S24–S29. Available at: https://journals.lww.com/alzheimerjournal/Fulltext/2010/07001/Barriers_and_Facilitators_of_African_American.6.aspx (accessed April 2019).10.1097/WAD.0b013e3181f14a14293913820711059Search in Google Scholar

Willis, G.B. 2005. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.10.1037/e538062007-001Search in Google Scholar

Willis, G.B. and K. Miller. 2011. “Cross-Cultural Cognitive Interviewing: Seeking Comparability and Enhancing Understanding.” Field Methods 23: 331–341. Doi: https://doi.org/10.1177/1525822x11416092.10.1177/1525822x11416092Open DOISearch in Google Scholar

Willits, F.K., G.L. Theodori, and A.E. Luloff. 2016. “Another Look at Likert Scales.” Journal of Rural Social Sciences 31: 126–139. Available at: http://journalofruralsocialsciences.org/pages/Articles/JRSS%202016%2031/3/JRSS%202016%2031%203%20126-139.pdf (accessed April 2019).Search in Google Scholar

Yan, T. and R. Tourangeau. 2008. “Fast Times and Easy Questions: The Effects of Age, Experience and Question Complexity on Web Survey Response Times.” Applied Cognitive Psychology 22: 51–68. Available at: https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.1331 (accessed April 2019).10.1002/acp.1331(accessed2019)Open DOISearch in Google Scholar

Zheng, B., M.A. Hall, E. Dugan, K.E. Kidd, and D. Levine. 2002. “Development of a Scale to Measure Patients’ Trust in Health Insurers.” Health Services Research 37: 185–200. Doi: https://doi.org/10.1111/1475-6773.00145.10.1111/1475-6773.00145Open DOISearch in Google Scholar