The split-ballot multitrait-multimethod (SB-MTMM) approach has been used to evaluate the measurement quality of questions in survey research. It aims to reduce the response burden of the classic MTMM design, which requires repeating alternative formulations of a survey measure to the same respondent at least three times, by using combinations of two methods in multiple groups. The SB-MTMM approach has been applied to the European Social Survey (ESS) to examine the quality of questions across countries, including the differences in response design and measurement errors. Despite wide application of the SB-MTMM design in Europe, it is yet unknown whether the same quality of survey instruments can be achieved in both a different cultural context and in a logographic writing system, like the one in Taiwan.
This study tests for measurement invariance and compares measurement quality in Taiwan and Denmark, by estimating the reliability and validity of different response scales using the SB-MTMM approach. By using the same questions as in the ESS, a cross-cultural comparison is made, in order to understand whether the studied response scales perform equally well in Taiwan, compared to a European country. Results show that quality estimates are comparable across countries.
If the inline PDF is not rendering correctly, you can download the PDF file here.
Allison, P.D. 1987. “Estimation of Linear Models with Incomplete Data.” In Sociological Methodology, edited by C.C. Clogg, 71–103. Washington DC: American Sociological Association. Doi: https://doi.org/10.2307/271029.
Alwin, D.F. 1974. “Approaches to the Interpretation of Relationships in the Multitrait Multimethod Matrix.” Sociological Methodology 5: 79–105. Doi: https://doi.org/10.2307/270833.
Alwin, D.F. and J.A. Krosnick. 1991. “The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes.” Sociological Methods and Research 20: 139–181. Doi: https://doi.org/10.1177/0049124191020001005.
Butts, M.M., R.J. Vandenberg, and L.J. Williams. 2006. “Investigating the Susceptibility of Measurement Invariance Tests: The Effects of Common Method Variance.” Academy of Management Proceedings 2006(1): D1–D6. Doi: https://doi.org/10.5465/AMBPp.2006.27182126.
Fu, Y.-C., Y.-H. Chang, S.-H. Tu, and P.-S. Liao. 2016. 2015 Taiwan Social Change Survey (Round 7, Year 1): Globalization, Work, Family, Mental Health, and Political Participation (C00315_2) [Data file]. Available at Survey Research Data Archive, Academia Sinica. Doi: https://doi.org/10.6141/TW-SRDA-C00315_1-1.
Goerman, P.L. and R.A. Caspar. 2010. “Managing the Cognitive Pretesting of Multilingual Survey Instruments: A Case Study of Pretesting of the U.S. Census Bureau Bilingual Spanish/English Questionnaire.” In Survey Methods in Multinational, Multiregional, and Multicultural Contexts, edited by J. Harkness, et al.: 75–90. John Wiley and Sons, Inc. Doi: https://doi.org/10.1002/9780470609927.ch5.
Harzing, A.W. 2006. “Response Styles in Cross-National Survey Research: A 26 Country Study.” International Journal of Cross Cultural Management 6 (2)(August 1): 243–266. Doi: https://doi.org/10.1177/1470595806066332.
Holbrook, A., Y.K. Cho, and T. Johnson. 2006. “The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties.” Public Opinion Quarterly 70: 565–595. Doi: https://doi.org/10.1093/poq/nfl027.
Horn, J.L. and J.J. McArdle. 1992. “A Practical and Theoretical Guide to Measurement Invariance in Aging Research.” Experimental Aging Research 18(3–4): 117–144. Doi: https://doi.org/10.1080/03610739208253916.
Lau, C.Q. 2016. “Rating Scale Design among Ethiopian Entrepreneurs: A Split-Ballot Experiment.” International Journal of Public Opinion Research edw031. Doi: https://doi.org/10.1093/ijpor/edw031.
Liao, P.-S. 2014. “More Happy or Less Unhappy? Comparison of the Balanced and Unbalanced Designs for the Response Scale of General Happiness.” Journal of Happiness Studies 15(6): 1407–1423. Doi: https://doi.org/10.1007/s10902-013-9484-1.
Marsh, H.W. and B.M. Byrne. 1993. “Confirmatory Factor Analysis of Multitrait-Multimethod Self-concept Data: Between-group and Within-group Invariance Constraints.” Multivariate Behavior Research 28(3): 313–449. Doi: https://doi.org/10.1207/s15327906mbr2803_2.
Van Meurs, A. and W.E. Saris. 1990. “Memory Effects in MTMM Studies.” In Evaluations of Measurement Instruments by Metaanalysis of Multitrait-Multimethod Studies, edited by W.E. Saris and A. van Meurs, 134–146. Amsterdam: North Holland.
Oberski, D., W.E. Saris, and J.A. Hagenaars. 2010. “Categorization Errors and Differences in the Quality of Questions in Comparative Surveys.” In Survey Methods in Multinational, Multiregional, and Multicultural Contexts, edited by J. Harkness, et al.: 435–453. Hoboken, NJ: Wiley. Doi: https://doi.org/10.1002/9780470609927.ch23.
Pan, Y., B. Craig, and S. Scollon. 2005. “Results from Chinese Cognitive Interviews on the Census 2000 Long Form: Language, Literacy, and Cultural Issues.” Statistical Research Division’s Research Report Series (Survey Methodology 2005 – 09). Washington, DC: U.S. Bureau of the Census. Available at https://www.census.gov/srd/papers/pdf/rsm2005-09.pdf (accessed November 2017).
Revilla, M. 2015. “Comparison of the Quality Estimates in a Mixed-Mode and a Unimode Design: An Experiment from the European Social Survey.” Quality and Quantity 49(3): 1219–1238. Doi: https://doi.org/10.1007/s11135-014-0044-5.
Revilla, M. and W.E. Saris. 2013. “The Split-Ballot Multitrait-Multimethod Approach: Implementation and Problems.” Structural Equation Modeling: A Multidisciplinary Journal 20: 27–46. Doi: https://doi.org/10.1080/10705511.2013.742379.
Revilla, M., W.E. Saris, G. Loewe, and C. Ochoa. 2015. “Can a Non-Probabilistic Online Panel Get Similar Question Quality as the ESS?” International Journal of Market Research 57(3): 395–412. Available at: https://www.mrs.org.uk/ijmr_article/article/104501 (accessed January 2019).
Saris, W.E. and F.M. Andrews. 1991. “Evaluation of Measurement Instruments Using a Structural Modeling Approach.” In Measurement Errors in Surveys, edited by P.P. Biemer, et al.: 575–597. New York, NY: Wiley.
Saris, W.E. and I.N. Gallhofer. 2007. “Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions.” Survey Research Methods 1: 29–43. Doi: http://dx.doi.org/10.18148/srm/2007.v1i1.49.
Saris, W.E. and I.N. Gallhofer. 2014. Design, Evaluation, and Analysis of Questionnaires for Survey Research (Second edition). Hoboken, NJ: Wiley.
Saris, W.E., A. Satorra, and G. Coenders. 2004. “A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design.” Sociological Methodology 34: 311–347. Doi: https://doi.org/10.1111/j.0081-1750.2004.00155.x.
Saris, W.E., A. Satorra, and W.M. van der Veld. 2009. “Testing Structural Equation Models or Detection of Misspecifications?” Structural Equation Modeling: A Multidisciplinary Journal 16(4): 561–582. Doi: https://doi.org/10.1080/10705510903203433.
Saris, W.E., R. Veenhoven, A.C. Scherpenzeel, and B. Brunting. 2008. A Comparative Study of Satisfaction with Life in Europe. Budapest: Eötvös University Press.
Saris, W.E., M. Revilla, J.A. Krosnick, and E.M. Shaffer. 2010. “Comparing Questions with Agree/Disagree Response Options to Questions with Item-Specific Response Options.” Survey Research Methods 4: 61–79. Doi: https://doi.org/10.18148/srm/2010.v4i1.2682.
Saris, W., D. Oberski, M. Revilla, D. Zavala, L. Lilleoja, I. Gallhofer, and T. Gruner. 2011. “The Development of the Program SQP 2.0 for the Prediction of the Quality of Survey Questions.” RECSM Working Paper 24, Universitat Pompeu Fabra. Available at: https://www.upf.edu/documents/3966940/3986764/RECSM_wp024.pdf (accessed January 2019).
Satorra, A. 1993. “Asymptotic Robust Inferences in Multi-sample Analysis of Augmented Moment Matrices.” In Multivariate Analysis; Future Directions, edited by R. Rao and C.M. Cuadras, 211–229. Amsterdam, North Holland.
Schuman, H. and S. Presser. 1981. Questions and Answers in Attitude Surveys. New York: Academic Press.
Van der Veld, W., W.E. Saris, and A. Satorra. 2008. Jrule 2.0: User Manual, Unpublished document.
Weng, L-J. 2004. “Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability.” Educational and Psychological Measurement 64: 956–972. Doi: https://doi.org/10.1177/0013164404268674.
Wu, C-E. and Y-T. Lin. 2013. “Cross-Strait Economic Openness, Identity, and Vote Choice: An Analysis of the 2008 and 2012 Presidential Elections.” Journal of Electoral Studies 20(2): 1–36. Doi: https://doi.org/10.6612/tjes.2013.20.02.01-35.
Zavala-Rojas, D., R. Tormos, W. Weber, and M. Revilla. 2018. “Designing Response Scales with Multi-Trait-Multi-Method Experiments.” Mathematical Population Studies 25(2): 66–81. Doi: https://doi.org/10.1080/08898480.2018.1439241.