Cross-National Comparison of Equivalence and Measurement Quality of Response Scales in Denmark and Taiwan

Pei-shan Liao 1 , Willem E. Saris 2 ,  and Diana Zavala-Rojas 3
  • 1 Center for Survey Research, RCHSS, , 11529, Taipei
  • 2 Sociometric Research Foundation, , 08019, Barcelona, Spain
  • 3 RECSM, Universitat Pompeu Fabra, , 08005, Barcelona, Spain


The split-ballot multitrait-multimethod (SB-MTMM) approach has been used to evaluate the measurement quality of questions in survey research. It aims to reduce the response burden of the classic MTMM design, which requires repeating alternative formulations of a survey measure to the same respondent at least three times, by using combinations of two methods in multiple groups. The SB-MTMM approach has been applied to the European Social Survey (ESS) to examine the quality of questions across countries, including the differences in response design and measurement errors. Despite wide application of the SB-MTMM design in Europe, it is yet unknown whether the same quality of survey instruments can be achieved in both a different cultural context and in a logographic writing system, like the one in Taiwan.

This study tests for measurement invariance and compares measurement quality in Taiwan and Denmark, by estimating the reliability and validity of different response scales using the SB-MTMM approach. By using the same questions as in the ESS, a cross-cultural comparison is made, in order to understand whether the studied response scales perform equally well in Taiwan, compared to a European country. Results show that quality estimates are comparable across countries.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Allison, P.D. 1987. “Estimation of Linear Models with Incomplete Data.” In Sociological Methodology, edited by C.C. Clogg, 71–103. Washington DC: American Sociological Association. Doi:

  • Alwin, D.F. 1974. “Approaches to the Interpretation of Relationships in the Multitrait Multimethod Matrix.” Sociological Methodology 5: 79–105. Doi:

  • Alwin, D.F. 1997. “Feeling Thermometers versus 7-point Scales: Which Are Better?” Sociological Methods and Research 25: 318–340. Doi:

  • Alwin, D.F. and J.A. Krosnick. 1991. “The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes.” Sociological Methods and Research 20: 139–181. Doi:

  • Bjørnskov, C. 2010. “How Comparable Are The Gallup World Poll Life Satisfaction Data?” Journal of Happiness Studies 11: 41–60. Doi:

  • Butts, M.M., R.J. Vandenberg, and L.J. Williams. 2006. “Investigating the Susceptibility of Measurement Invariance Tests: The Effects of Common Method Variance.” Academy of Management Proceedings 2006(1): D1–D6. Doi:

  • Byrne, B.M. and D. Watkins. 2003. “The Issue of Measurement Invariance Revisited.” Journal of Cross-Cultural Psychology 34(2): 155–175. Doi:

  • Campbell, D.T. and D.W. Fiske. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.” Psychological Bulletin 56(2): 81–105. Doi:

  • Chen, C.K. 2005. “Construct Model of Knowledge: Based Economy Indicators.” Management Review 24(3): 17–41. Doi:

  • Chen, C., S.Y. Lee, and H.W. Stevenson. 1995. “Response Style and Cross-Cultural Comparisons of Rating Scales among East Asian and North American Students.” Psychological Science 6: 170–175. Doi:

  • ESS Round 1: European Social Survey. 2014. ESS-1 2002 Documentation Report. Edition 6.4. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC. Available at: (accessed May 2016).

  • Fu, Y.-C., Y.-H. Chang, S.-H. Tu, and P.-S. Liao. 2016. 2015 Taiwan Social Change Survey (Round 7, Year 1): Globalization, Work, Family, Mental Health, and Political Participation (C00315_2) [Data file]. Available at Survey Research Data Archive, Academia Sinica. Doi:

  • Goerman, P.L. and R.A. Caspar. 2010. “Managing the Cognitive Pretesting of Multilingual Survey Instruments: A Case Study of Pretesting of the U.S. Census Bureau Bilingual Spanish/English Questionnaire.” In Survey Methods in Multinational, Multiregional, and Multicultural Contexts, edited by J. Harkness, et al.: 75–90. John Wiley and Sons, Inc. Doi:

  • Harzing, A.W. 2006. “Response Styles in Cross-National Survey Research: A 26 Country Study.” International Journal of Cross Cultural Management 6 (2)(August 1): 243–266. Doi:

  • Hsiao, C.-C. and C.-H. Tu. 2012. “Common Method Variance in the Measurement of Teachers’ Creative Teaching.” Psychological Testing 59(4): 609–639. Doi:

  • Holbrook, A., Y.K. Cho, and T. Johnson. 2006. “The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties.” Public Opinion Quarterly 70: 565–595. Doi:

  • Horn, J.L. and J.J. McArdle. 1992. “A Practical and Theoretical Guide to Measurement Invariance in Aging Research.” Experimental Aging Research 18(3–4): 117–144. Doi:

  • Lau, C.Q. 2016. “Rating Scale Design among Ethiopian Entrepreneurs: A Split-Ballot Experiment.” International Journal of Public Opinion Research edw031. Doi:

  • Liao, P.-S. 2014. “More Happy or Less Unhappy? Comparison of the Balanced and Unbalanced Designs for the Response Scale of General Happiness.” Journal of Happiness Studies 15(6): 1407–1423. Doi:

  • Marsh, H.W. and B.M. Byrne. 1993. “Confirmatory Factor Analysis of Multitrait-Multimethod Self-concept Data: Between-group and Within-group Invariance Constraints.” Multivariate Behavior Research 28(3): 313–449. Doi:

  • Van Meurs, A. and W.E. Saris. 1990. “Memory Effects in MTMM Studies.” In Evaluations of Measurement Instruments by Metaanalysis of Multitrait-Multimethod Studies, edited by W.E. Saris and A. van Meurs, 134–146. Amsterdam: North Holland.

  • Oberski, D.L., W.E. Saris, and J. Hagenaars. 2007. “Why Are There Differences in Measurement Quality across Countries?” In Measuring Meaningful Data in Social Research, edited by G. Loosveldt and Swyngedouw. Leuven: Acco. Available at: (accessed January 2019).

  • Oberski, D., W.E. Saris, and J.A. Hagenaars. 2010. “Categorization Errors and Differences in the Quality of Questions in Comparative Surveys.” In Survey Methods in Multinational, Multiregional, and Multicultural Contexts, edited by J. Harkness, et al.: 435–453. Hoboken, NJ: Wiley. Doi:

  • Pan, Y., B. Craig, and S. Scollon. 2005. “Results from Chinese Cognitive Interviews on the Census 2000 Long Form: Language, Literacy, and Cultural Issues.” Statistical Research Division’s Research Report Series (Survey Methodology 2005 – 09). Washington, DC: U.S. Bureau of the Census. Available at (accessed November 2017).

  • Petersen, T. 2008. “Spilt Ballot as An Experimental Approach to Public Opinion Research.” In The Sage Handbook of Public Opinion Research, edited by W. Donsbach and M.W. Traugott, 322–329. Los Angeles, CA: Sage. Available at: (accessed January 2019).

  • Revilla, M. 2015. “Comparison of the Quality Estimates in a Mixed-Mode and a Unimode Design: An Experiment from the European Social Survey.” Quality and Quantity 49(3): 1219–1238. Doi:

  • Revilla, M. and W.E. Saris. 2013. “The Split-Ballot Multitrait-Multimethod Approach: Implementation and Problems.” Structural Equation Modeling: A Multidisciplinary Journal 20: 27–46. Doi:

  • Revilla, M., W.E. Saris, G. Loewe, and C. Ochoa. 2015. “Can a Non-Probabilistic Online Panel Get Similar Question Quality as the ESS?” International Journal of Market Research 57(3): 395–412. Available at: (accessed January 2019).

  • Saris, W.E. and F.M. Andrews. 1991. “Evaluation of Measurement Instruments Using a Structural Modeling Approach.” In Measurement Errors in Surveys, edited by P.P. Biemer, et al.: 575–597. New York, NY: Wiley.

  • Saris, W.E. and I.N. Gallhofer. 2007. “Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions.” Survey Research Methods 1: 29–43. Doi:

  • Saris, W.E. and I.N. Gallhofer. 2014. Design, Evaluation, and Analysis of Questionnaires for Survey Research (Second edition). Hoboken, NJ: Wiley.

  • Saris, W.E., A. Satorra, and G. Coenders. 2004. “A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design.” Sociological Methodology 34: 311–347. Doi:

  • Saris, W.E., A. Satorra, and W.M. van der Veld. 2009. “Testing Structural Equation Models or Detection of Misspecifications?” Structural Equation Modeling: A Multidisciplinary Journal 16(4): 561–582. Doi:

  • Saris, W.E., R. Veenhoven, A.C. Scherpenzeel, and B. Brunting. 2008. A Comparative Study of Satisfaction with Life in Europe. Budapest: Eötvös University Press.

  • Saris, W.E., M. Revilla, J.A. Krosnick, and E.M. Shaffer. 2010. “Comparing Questions with Agree/Disagree Response Options to Questions with Item-Specific Response Options.” Survey Research Methods 4: 61–79. Doi:

  • Saris, W., D. Oberski, M. Revilla, D. Zavala, L. Lilleoja, I. Gallhofer, and T. Gruner. 2011. “The Development of the Program SQP 2.0 for the Prediction of the Quality of Survey Questions.” RECSM Working Paper 24, Universitat Pompeu Fabra. Available at: (accessed January 2019).

  • Satorra, A. 1993. “Asymptotic Robust Inferences in Multi-sample Analysis of Augmented Moment Matrices.” In Multivariate Analysis; Future Directions, edited by R. Rao and C.M. Cuadras, 211–229. Amsterdam, North Holland.

  • Schaeffer, N.C. and S. Presser. 2003. “The Science of Asking Questions.” Annual Review of Sociology 29: 65–88. Doi:

  • Schuman, H. and S. Presser. 1981. Questions and Answers in Attitude Surveys. New York: Academic Press.

  • Van der Veld, W., W.E. Saris, and A. Satorra. 2008. Jrule 2.0: User Manual, Unpublished document.

  • Weng, L-J. 2004. “Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability.” Educational and Psychological Measurement 64: 956–972. Doi:

  • Wu, C-E. and Y-T. Lin. 2013. “Cross-Strait Economic Openness, Identity, and Vote Choice: An Analysis of the 2008 and 2012 Presidential Elections.” Journal of Electoral Studies 20(2): 1–36. Doi:

  • Zavala-Rojas, D., R. Tormos, W. Weber, and M. Revilla. 2018. “Designing Response Scales with Multi-Trait-Multi-Method Experiments.” Mathematical Population Studies 25(2): 66–81. Doi:


Journal + Issues