Effects of Scale Direction on Response Style of Ordinal Rating Scales

Open access


Although ordinal rating scales have received much research attention in survey methodology literature, the direction of the rating scales has not been as extensively studied as other design features. Research on scale direction effect has mainly focused on the influence on response distribution, while largely overlooking its impact on latent constructs. This study examines the scale direction effect on extreme and acquiescent response style latent class variables in an experiment embedded in a national probability sample. We found a higher level of acquiescent response style from scales starting with positive adjective words using a web survey. No significant effect of scale direction was detected on extreme response style or in a face-to-face survey (with show cards). This study also demonstrates that scale direction does not affect the substance latent class variables, once the response style latent class variables are included in the model. Implications of these findings and future research directions are discussed.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Baumgartner H. and J.E.M. Steenkamp. 2001. “Response Styles in Marketing Research: A Cross-National Investigation.” Journal of Marketing Research 38: 143-156. Doi: http://dx.doi.org/10.1509/jmkr.

  • Belson W.A. 1966. “Effects of Reversing Presentation Order of Verbal Rating Scales.” Journal of Advertising Research 6: 30-37.

  • Carp F.M. 1974. “Position Effects on Interview Responses.” Journal of Gerontology 29: 581-587. Doi: http://dx.doi.org/10.1093/geronj/29.5.581.

  • Chan J.C. 1991. “Response-Order Effect in Likert-Type Scales.” Educational and Psychological Measurement 51: 531-540. Doi: http://dx.doi.org/10.1177/0013164491513002.

  • Christian L.M. D.A. Dillman and J.D. Smyth. 2008. “The Effects of Mode and Format on Answers to Scalar Questions in Telephone and Web Surveys.” In Advances in Telephone Survey Methodology edited by J.M. Lepkowski C. Tucker J.M. Brick E.D. De Leeuw L. Japec P.J. LavrakasM.W.Link R.L. Sangster 250-275.New Jersey: JohnWiley&Sons Inc.

  • Dillman D.A. T.L. Brown J.E. Carlson E.H. Carpenter F.O. Lorenz R. Mason J. Saltiel and R.L. Songster. 1995. “Effects of Category Order on Answers in Mail and Telephone Surveys.” Rural Sociology 60: 674-687. Doi: http://dx.doi.org/10.1111/j. 1549-0831.1995.tb00600.x.

  • He L. T. Yan F. Keusch and S. Han. 2014. “The impact of question and scale characteristics on scale direction effect.” In Proccedings of the AAPOR 69th Annual Conference Anaheim California May 15-18. Available at: http://www.aapor.org/AAPOR_Main/media/AnnualMeetingProceedings/2014/Session-J-6-2-He-Y.pdf (accessed January 2017).

  • Kalton G. M. Collins and L. Brook. 1978. “Experiments in Wording Opinion Questions.” Applied Statistics 149-161. Doi: http://dx.doi.org/10.2307/2346942.

  • Kieruj N.D. and G. Moors. 2010. “Variations in Response Style Behavior by Response Scale Format in Attitude Research.” International Journal of Public Opinion Research 22: 320-342. Doi: http://dx.doi.org/10.1093/ijpor/edq001.

  • Kieruj N.D. and G. Moors. 2013. “Response Style Behavior: Question Format Dependent or Personal Style?” Quality & Quantity 47: 193-211. Doi: http://dx.doi.org/10.1007/s11135-011-9511-4.

  • Krebs D. and J.H.P. Hoffmeyer-Zlotnik. 2010. “Positive First or Negative First?: Effects of the Order of Answering Categories on Response Behavior.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 6: 118-127. Doi: http://dx.doi.org/10.1027/1614-2241/a000013.

  • Krosnick J.A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5: 213-236. Doi: http://dx.doi.org/10.1002/acp.2350050305.

  • Krosnick J.A. and S. Presser. 2010. “Question and Questionnaire Design.” In Handbook of Survey Research Second Edition edited by P.V. Marsden and J.D. Wright 263-314. United Kingdom: Emerald Group Publishing Limited.

  • Liu M. F.G. Conrad and S. Lee. 2016. “Comparing Acquiescent and Extreme Response Styles in Face-to-Face and Web Surveys.” Quality & Quantity. Doi: http://dx.doi.org/10.1007/s11135-016-0320-7.

  • Liu M. S. Lee and F.G. Conrad. 2015. “Comparing Extreme Response Styles between Agree-Disagree and Item-Specific Scales.” Public Opinion Quarterly 79: 952-975. Doi: http://dx.doi.org/10.1093/poq/nfv034.

  • Mingay D.J. and M.T. Greenwell. 1989. “Memory Bias and Response-Order Effects.” Journal of Official Statistics 5: 253-263.

  • Moors G. 2003. “Diagnosing Response Style Behavior by Means of a Latent-Class Factor Approach. Socio-Demographic Correlates of Gender Role Attitudes and Perceptions of Ethnic Discrimination Reexamined.” Quality and Quantity 37: 277-302. Doi: http://dx.doi.org/10.1023/A:1024472110002.

  • Moors G. 2008. “Exploring the Effect of a Middle Response Category on Response Style in Attitude Measurement.” Quality & Quantity 42: 779-794. Doi: http://dx.doi.org/10.1007/s11135-006-9067-x.

  • Moors G. N.D. Kieruj and J.K. Vermunt. 2014. “The Effect of Labeling and Numbering of Response Scales on the Likelihood of Response Bias.” Sociological Methodology 44: 369-399. Doi: http://dx.doi.org/10.1177/0081175013516114.

  • Morren M. J.P.T.M. Gelissen and J.K. Vermunt. 2011. “Dealing With Extreme Response Style In Cross-Cultural Research: A Restricted Latent Class Factor Analysis Approach: Extreme Response Style In Cross-Cultural Research.” Sociological Methodology 41: 13-47. Doi: http://dx.doi.org/10.1111/j.1467-9531.2011.01238.x.

  • Paulhus D.L. 1991. “Measurement and Control of Response Bias.” Measures of Personality and Social Psychological Attitudes. Volume 1 in Measures of Social Psychological Attitudes Series. Available at: http://doi.apa.org/psycinfo/1991-97206-001.

  • Revilla M.A. W.E. Saris and J.A. Krosnick. 2013. “Choosing the Number of Categories in Agree-Disagree Scales.” Sociological Methods & Research 43: 73-97. Doi: http://dx.doi.org/10.1177/0049124113509605.

  • Salzberger T. and M. Koller. 2013. “Towards a New Paradigm of Measurement in Marketing.” Journal of Business Research 66: 1307-1317.

  • Saris W.E. and I.N. Gallhofer. 2007. “Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions.” In Design Evaluation and Analysis of Questionnaires for Survey Research edited by W.E. Saris and I.N. Gallhofer 237-253. New Jersey: John Wiley & Sons Inc.

  • Stapleton C. 2013. “The Smart (phone)Way to Collect Survey Data.” Survey Practice 6(2).

  • Tourangeau R. M.P. Couper and F.G. Conrad. 2004. “Spacing Position and Order Interpretive Heuristics for Visual Features of Survey Questions.” Public Opinion Quarterly 68: 368-393. Doi: http://dx.doi.org/10.1093/poq/nfh035.

  • Tourangeau R. M.P. Couper and F.G. Conrad. 2007. “Color Labels and Interpretive Heuristics for Response Scales.” Public Opinion Quarterly 71: 91-112. Doi: http://dx.doi.org/10.1093/poq/nfl046.

  • Tversky A. and D. Kahneman. 1974. “Judgment under Uncertainty: Heuristics and Biases.” Science 185: 1124-1131. Doi: http://dx.doi.org/10.1126/science.185.4157.1124.

  • Vermunt J.K. and J. Magidson. 2013. Technical guide for Latent GOLD 5.0: Basic advanced and syntax. Belmont MA: Statistical Innovations Inc.

  • Yan T. and F. Keusch. 2015. “The Effects of the Direction of Rating Scales on Survey Responses in a Telephone Survey.” Public Opinion Quarterly 79: 145-165. Doi: http://dx.doi.org/10.1093/poq/nfu062.

Journal information
Impact Factor

IMPACT FACTOR 2018: 0.837
5-year IMPACT FACTOR: 0.934

CiteScore 2018: 1.04

SCImago Journal Rank (SJR) 2018: 0.963
Source Normalized Impact per Paper (SNIP) 2018: 1.020

Cited By
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 603 400 44
PDF Downloads 356 270 34