Cross-National Comparison of Equivalence and Measurement Quality of Response Scales in Denmark and Taiwan

Open access


The split-ballot multitrait-multimethod (SB-MTMM) approach has been used to evaluate the measurement quality of questions in survey research. It aims to reduce the response burden of the classic MTMM design, which requires repeating alternative formulations of a survey measure to the same respondent at least three times, by using combinations of two methods in multiple groups. The SB-MTMM approach has been applied to the European Social Survey (ESS) to examine the quality of questions across countries, including the differences in response design and measurement errors. Despite wide application of the SB-MTMM design in Europe, it is yet unknown whether the same quality of survey instruments can be achieved in both a different cultural context and in a logographic writing system, like the one in Taiwan.

This study tests for measurement invariance and compares measurement quality in Taiwan and Denmark, by estimating the reliability and validity of different response scales using the SB-MTMM approach. By using the same questions as in the ESS, a cross-cultural comparison is made, in order to understand whether the studied response scales perform equally well in Taiwan, compared to a European country. Results show that quality estimates are comparable across countries.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Allison P.D. 1987. “Estimation of Linear Models with Incomplete Data.” In Sociological Methodology edited by C.C. Clogg 71–103. Washington DC: American Sociological Association. Doi:

  • Alwin D.F. 1974. “Approaches to the Interpretation of Relationships in the Multitrait Multimethod Matrix.” Sociological Methodology 5: 79–105. Doi:

  • Alwin D.F. 1997. “Feeling Thermometers versus 7-point Scales: Which Are Better?” Sociological Methods and Research 25: 318–340. Doi:

  • Alwin D.F. and J.A. Krosnick. 1991. “The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes.” Sociological Methods and Research 20: 139–181. Doi:

  • Bjørnskov C. 2010. “How Comparable Are The Gallup World Poll Life Satisfaction Data?” Journal of Happiness Studies 11: 41–60. Doi:

  • Butts M.M. R.J. Vandenberg and L.J. Williams. 2006. “Investigating the Susceptibility of Measurement Invariance Tests: The Effects of Common Method Variance.” Academy of Management Proceedings 2006(1): D1–D6. Doi:

  • Byrne B.M. and D. Watkins. 2003. “The Issue of Measurement Invariance Revisited.” Journal of Cross-Cultural Psychology 34(2): 155–175. Doi:

  • Campbell D.T. and D.W. Fiske. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.” Psychological Bulletin 56(2): 81–105. Doi:

  • Chen C.K. 2005. “Construct Model of Knowledge: Based Economy Indicators.” Management Review 24(3): 17–41. Doi:

  • Chen C. S.Y. Lee and H.W. Stevenson. 1995. “Response Style and Cross-Cultural Comparisons of Rating Scales among East Asian and North American Students.” Psychological Science 6: 170–175. Doi:

  • ESS Round 1: European Social Survey. 2014. ESS-1 2002 Documentation Report. Edition 6.4. Bergen European Social Survey Data Archive NSD – Norwegian Centre for Research Data for ESS ERIC. Available at: (accessed May 2016).

  • Fu Y.-C. Y.-H. Chang S.-H. Tu and P.-S. Liao. 2016. 2015 Taiwan Social Change Survey (Round 7 Year 1): Globalization Work Family Mental Health and Political Participation (C00315_2) [Data file]. Available at Survey Research Data Archive Academia Sinica. Doi:

  • Goerman P.L. and R.A. Caspar. 2010. “Managing the Cognitive Pretesting of Multilingual Survey Instruments: A Case Study of Pretesting of the U.S. Census Bureau Bilingual Spanish/English Questionnaire.” In Survey Methods in Multinational Multiregional and Multicultural Contexts edited by J. Harkness et al.: 75–90. John Wiley and Sons Inc. Doi:

  • Harzing A.W. 2006. “Response Styles in Cross-National Survey Research: A 26 Country Study.” International Journal of Cross Cultural Management 6 (2)(August 1): 243–266. Doi:

  • Hsiao C.-C. and C.-H. Tu. 2012. “Common Method Variance in the Measurement of Teachers’ Creative Teaching.” Psychological Testing 59(4): 609–639. Doi:

  • Holbrook A. Y.K. Cho and T. Johnson. 2006. “The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties.” Public Opinion Quarterly 70: 565–595. Doi:

  • Horn J.L. and J.J. McArdle. 1992. “A Practical and Theoretical Guide to Measurement Invariance in Aging Research.” Experimental Aging Research 18(3–4): 117–144. Doi:

  • Lau C.Q. 2016. “Rating Scale Design among Ethiopian Entrepreneurs: A Split-Ballot Experiment.” International Journal of Public Opinion Research edw031. Doi:

  • Liao P.-S. 2014. “More Happy or Less Unhappy? Comparison of the Balanced and Unbalanced Designs for the Response Scale of General Happiness.” Journal of Happiness Studies 15(6): 1407–1423. Doi:

  • Marsh H.W. and B.M. Byrne. 1993. “Confirmatory Factor Analysis of Multitrait-Multimethod Self-concept Data: Between-group and Within-group Invariance Constraints.” Multivariate Behavior Research 28(3): 313–449. Doi:

  • Van Meurs A. and W.E. Saris. 1990. “Memory Effects in MTMM Studies.” In Evaluations of Measurement Instruments by Metaanalysis of Multitrait-Multimethod Studies edited by W.E. Saris and A. van Meurs 134–146. Amsterdam: North Holland.

  • Oberski D.L. W.E. Saris and J. Hagenaars. 2007. “Why Are There Differences in Measurement Quality across Countries?” In Measuring Meaningful Data in Social Research edited by G. Loosveldt and Swyngedouw. Leuven: Acco. Available at: (accessed January 2019).

  • Oberski D. W.E. Saris and J.A. Hagenaars. 2010. “Categorization Errors and Differences in the Quality of Questions in Comparative Surveys.” In Survey Methods in Multinational Multiregional and Multicultural Contexts edited by J. Harkness et al.: 435–453. Hoboken NJ: Wiley. Doi:

  • Pan Y. B. Craig and S. Scollon. 2005. “Results from Chinese Cognitive Interviews on the Census 2000 Long Form: Language Literacy and Cultural Issues.” Statistical Research Division’s Research Report Series (Survey Methodology 2005 – 09). Washington DC: U.S. Bureau of the Census. Available at (accessed November 2017).

  • Petersen T. 2008. “Spilt Ballot as An Experimental Approach to Public Opinion Research.” In The Sage Handbook of Public Opinion Research edited by W. Donsbach and M.W. Traugott 322–329. Los Angeles CA: Sage. Available at: (accessed January 2019).

  • Revilla M. 2015. “Comparison of the Quality Estimates in a Mixed-Mode and a Unimode Design: An Experiment from the European Social Survey.” Quality and Quantity 49(3): 1219–1238. Doi:

  • Revilla M. and W.E. Saris. 2013. “The Split-Ballot Multitrait-Multimethod Approach: Implementation and Problems.” Structural Equation Modeling: A Multidisciplinary Journal 20: 27–46. Doi:

  • Revilla M. W.E. Saris G. Loewe and C. Ochoa. 2015. “Can a Non-Probabilistic Online Panel Get Similar Question Quality as the ESS?” International Journal of Market Research 57(3): 395–412. Available at: (accessed January 2019).

  • Saris W.E. and F.M. Andrews. 1991. “Evaluation of Measurement Instruments Using a Structural Modeling Approach.” In Measurement Errors in Surveys edited by P.P. Biemer et al.: 575–597. New York NY: Wiley.

  • Saris W.E. and I.N. Gallhofer. 2007. “Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions.” Survey Research Methods 1: 29–43. Doi:

  • Saris W.E. and I.N. Gallhofer. 2014. Design Evaluation and Analysis of Questionnaires for Survey Research (Second edition). Hoboken NJ: Wiley.

  • Saris W.E. A. Satorra and G. Coenders. 2004. “A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design.” Sociological Methodology 34: 311–347. Doi:

  • Saris W.E. A. Satorra and W.M. van der Veld. 2009. “Testing Structural Equation Models or Detection of Misspecifications?” Structural Equation Modeling: A Multidisciplinary Journal 16(4): 561–582. Doi:

  • Saris W.E. R. Veenhoven A.C. Scherpenzeel and B. Brunting. 2008. A Comparative Study of Satisfaction with Life in Europe. Budapest: Eötvös University Press.

  • Saris W.E. M. Revilla J.A. Krosnick and E.M. Shaffer. 2010. “Comparing Questions with Agree/Disagree Response Options to Questions with Item-Specific Response Options.” Survey Research Methods 4: 61–79. Doi:

  • Saris W. D. Oberski M. Revilla D. Zavala L. Lilleoja I. Gallhofer and T. Gruner. 2011. “The Development of the Program SQP 2.0 for the Prediction of the Quality of Survey Questions.” RECSM Working Paper 24 Universitat Pompeu Fabra. Available at: (accessed January 2019).

  • Satorra A. 1993. “Asymptotic Robust Inferences in Multi-sample Analysis of Augmented Moment Matrices.” In Multivariate Analysis; Future Directions edited by R. Rao and C.M. Cuadras 211–229. Amsterdam North Holland.

  • Schaeffer N.C. and S. Presser. 2003. “The Science of Asking Questions.” Annual Review of Sociology 29: 65–88. Doi:

  • Schuman H. and S. Presser. 1981. Questions and Answers in Attitude Surveys. New York: Academic Press.

  • Van der Veld W. W.E. Saris and A. Satorra. 2008. Jrule 2.0: User Manual Unpublished document.

  • Weng L-J. 2004. “Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability.” Educational and Psychological Measurement 64: 956–972. Doi:

  • Wu C-E. and Y-T. Lin. 2013. “Cross-Strait Economic Openness Identity and Vote Choice: An Analysis of the 2008 and 2012 Presidential Elections.” Journal of Electoral Studies 20(2): 1–36. Doi:

  • Zavala-Rojas D. R. Tormos W. Weber and M. Revilla. 2018. “Designing Response Scales with Multi-Trait-Multi-Method Experiments.” Mathematical Population Studies 25(2): 66–81. Doi:

Journal information
Impact Factor

IMPACT FACTOR 2018: 0.837
5-year IMPACT FACTOR: 0.934

CiteScore 2018: 1.04

SCImago Journal Rank (SJR) 2018: 0.963
Source Normalized Impact per Paper (SNIP) 2018: 1.020

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 356 356 10
PDF Downloads 198 198 8