Differences Across Levels in the Language of Agency and Ability in Rating Scales for Large-Scale Second Language Writing Assessments

Open access

Abstract

While large-scale language and writing assessments benefit from a wealth of literature on the reliability and validity of specific tests and rating procedures, there is comparatively less literature that explores the specific language of second language writing rubrics. This paper provides an analysis of the language of performance descriptors for the public versions of the TOEFL and IELTS writing assessment rubrics, with a focus on linguistic agency encoded by agentive verbs and language of ability encoded by modal verbs can and cannot. While the IELTS rubrics feature more agentive verbs than the TOEFL rubrics, both pairs of rubrics feature uneven syntax across the band or score descriptors with either more agentive verbs for the highest scores, more nominalization for the lowest scores, or language of ability exclusively in the lowest scores. These patterns mirror similar patterns in the language of college-level classroom-based writing rubrics, but they differ from patterns seen in performance descriptors for some large-scale admissions tests. It is argued that the lack of syntactic congruity across performance descriptors in the IELTS and TOEFL rubrics may reflect a bias in how actual student performances at different levels are characterized.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Ahearn Laura M. 2001. Language and agency. Annual Review of Anthropology 30. 109–137. DOI: 10.1146/annurev.anthro.30.1.109

  • Alderson J. Charles. 1991. Bands and scores. In J. Charles Alderson & Brian North (eds.) Language testing in the 1990s: The communicative legacy 71–86. London: Modern English Publications/British Council.

  • Alderson J. Charles Neus Figueras Henk Kuijper Guenter Nold Sauli Takala & Claire Tardieu. 2004. The development of specifications for item development and classification within the Common European Framework of Reference for Languages: Learning teaching assessment: Reading and listening: Final report of the Dutch CEF Construct Project. Lancaster University. http://eprints.lancs.ac.uk/44/1/final_report.pdf.

  • Bachman Lyle F. 1990. Fundamental considerations in language testing. Oxford: Oxford University Press.

  • Banerjee Jayanti Xun Yan Mark Chapman & Heather Elliott. 2015. Keeping up with the times: Revising and refreshing a rating scale. Assessing Writing 26. 5–19. DOI: 10.1016/j.asw.2015.07.001

  • Becker Anthony. 2010. Examining rubrics used to measure writing performance in US intensive English programs. The CATESOL Journal 22(1). 113–130.

  • Billig Michael. 2008. The language of critical discourse analysis: The case of nominalization. Discourse & Society 19(6). 783–800. DOI: 10.1177/0957926508095894

  • Brindley Geoff. 1998. Describing language development? Rating scales and SLA. In Lyle F. Bachman & Andrew D. Cohen (eds.) Interfaces between second language acquisition and language testing research 112–140. New York: Cambridge University Press.

  • Bucholtz Mary & Kira Hall. 2005. Identity and interaction: A sociocultural linguistic approach. Discourse Studies 7(4–5). 585–614. DOI: 10.1177/1461445605054407

  • Calkins Lucy McCormick. 1994. The art of teaching writing (new ed.). Portsmouth: Heinemann.

  • Chakroff Aleksandr Kyle A. Thomas Omar S. Haque & Liane Young. 2015. An indecent proposal: The dual functions of indirect speech. Cognitive Science 39(1). 199–211. DOI: 10.1111/cogs.12145

  • Cotton Fiona & Kate Wilson. 2011. An investigation of examiner rating of coherence and cohesion in the IELTS Academic Writing Task 2. https://www.ielts.org/-/media/research-reports/ielts_rr_volume12_report6.ashx

  • Covill Amy E. 2012. College students’ use of a writing rubric: Effect on quality of writing self-efficacy and writing practices. Journal of Writing Assessment 5(1). http://journalofwritingassessment.org/article.php?article=60

  • Crusan Deborah. 2010. Assessment in the second language writing classroom. Ann Arbor: University of Michigan Press.

  • Davies Alan. 2008. Assessing academic English. Testing English proficiency 1950–89: The IELTS solution. Cambridge: Cambridge University Press.

  • Dowty David. 1991. Thematic proto-roles and argument selection. Language 67(3). 547–619.

  • Dryer Dylan. 2013. Scaling writing ability: A corpus-driven inquiry. Written Communication 30(1). 3–35. DOI: 10.1177/0741088312466992

  • Duranti Alessandro. 2004. Agency in language. In Alessandro Duranti (ed.) A companion to linguistic anthropology 451–473. Malden MA: Blackwell. DOI: 10.1002/9780470996522.ch20

  • Ehrlich Susan. 2001. Representing rape: Language and sexual consent. New York: Routledge.

  • Fausey Caitlin M. & Lera Boroditsky. 2010. Subtle linguistic cues influence perceived blame and financial liability. Psychonomic Bulletin & Review 17(5). 644–650. DOI: 10.3758/PBR.17.5.644

  • Fausey Caitlin M. Bria L. Long Aya Inamori & Lera Boroditsky. 2010. Constructing agency: The role of language. Frontiers in Psychology 1. 162. DOI: 10.3389/fpsyg.2010.00162

  • Fillmore Charles J. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences 280. 20–32. DOI: 10.1111/j.1749-6632.1976.tb25467.x

  • Fillmore Charles J. & Collin Baker. 2010. A frames approach to semantic analysis. In Bernd Heine & Heiko Narrog (eds.) The Oxford handbook of linguistic analysis 313–339. Oxford: Oxford University Press. DOI: 10.1093/oxfordhb/9780199544004.013.0013

  • Fowler Roger Bob Hodge Günther Kress & Tony Trew. 1979. Language and control. London: Routledge.

  • Fowler Roger. 1991. Language in the news: Discourse and ideology in the press. London: Routledge.

  • Fox Janna D. 2007. Language testing reconsidered. Ottawa: University of Ottawa Press.

  • Golder Katherine Kenneth Reeder & Sarah Fleming. 2012. Determination of appropriate IELTS Writing and Speaking Band Scores for admission into two programs at a Canadian post-secondary polytechnic institution. Canadian Journal of Applied Linguistics/Revue canadienne de linguistique appliquée 14(1). 222–250.

  • Hambleton Ronald K.& Mary Pitoniak. 2006. Setting performance standards. In Robert L. Brennan (ed.) Educational measurement (4th ed.) 433–470. Westport CT: Praeger.

  • Hawkey Roger & Fiona Barker. 2004. Developing a common scale for the assessment of writing. Assessing Writing 9(2). 122–159. DOI: 10.1016/j.asw.2004.06.001

  • Henley Nancy M. Michelle Miller & Jo Anne Beazley. 1995. Syntax semantics and sexual violence: Agency and the passive voice. Journal of Language and Social Psychology 14(1–2). 60–84. DOI: 10.1177/0261927X95141004

  • Jeffery Jill V. 2009. Constructs of writing proficiency in US state and national writing assessments: Exploring variability. Assessing Writing 14(1). 3–24. DOI: 10.1016/j.asw.2008.12.002

  • Knoch Ute. 2007. ‘Little coherence considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing 12(2). 108–128. DOI: 10.1016/j.asw.2007.07.002

  • Knoch Ute. 2009. Diagnostic assessment of writing: A comparison of two rating scales. Language Testing 26(2). 275–304. DOI: 10.1177/0265532208101008

  • Knoch Ute. 2011. Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from? Assessing Writing 16(2). 81–96. DOI: 10.1016/j.asw.2011.02.003

  • Knoch Ute Susy Macqueen & Sally O'Hagan. 2014. An investigation of the effect of task type on the discourse produced by students at various score levels in the TOEFL iBT® writing test. ETS Research Report Series 2014(2). DOI: 10.1002/ets2.12038

  • Kuiken Folkert & Ineke Vedder. 2014. Raters’ decisions rating procedures and rating scales. Language Testing. 31(3). 279–284. DOI: 10.1177/0265532214526179

  • LaFrance Marianne & Eugene Hahn. 1994. The disappearing agent: Gender stereotypes interpersonal verbs and implicit causality. In Camille Roman Suzanne Juhasz & Cristianne Miller (eds.) The women and language debate: A sourcebook 348–362. New Brunswick NJ: Rutgers University Press.

  • Li Jinrong & Peggy Lindsey. 2015. Understanding variations between student and teacher application of rubrics. Assessing Writing 26. 67–79. DOI: 10.1016/j.asw.2015.07.003

  • Lumley Tom. 2002. Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing 19(3). 246–276. DOI:10.1191/0265532202lt230oa

  • Matsuda Paul Kei & Jill V. Jeffery. 2012. Voice in student essays. In Ken Hyland & Carmen Sancho Guinda (eds.) Stance and voice in written academic genres 151–165. New York: Palgrave Macmillan. DOI: 10.1057/9781137030825_10

  • Messick Samuel. 1988. Meaning and values in test validation: The science and ethics of assessment. ETS Research Report Series 1988(2). DOI:10.1002/j.2330-8516.1988.tb00303.x

  • Morales Meghan Corella & Jin Sook Lee. 2015. Stories of assessment: Spanish–English bilingual children's agency and interactional competence in oral language assessments. Linguistics and Education 29. 32–45. DOI: 10.1016/j.linged.2014.10.008

  • North Brian. 2007. The CEFR illustrative descriptor scales. The Modern Language Journal 91(4). 656–659. DOI: 10.1111/j.1540-4781.2007.00627_3.x

  • North Brian& Günther Schneider. 1998. Scaling descriptors for language proficiency scales. Language Testing 15(2). 217–262. DOI: 10.1177/026553229801500204

  • [OED =] Oxford English Dictionary (3rd ed.). Oxford: Oxford University Press. https://www.oed.com/

  • Schaefer Edward. 2008. Rater bias patterns in an EFL writing assessment. Language Testing 25(4). 465–493. DOI: 10.1177/0265532208094273

  • Spandel Vicki. 2006. In defense of rubrics. English Journal 96(1). 19–22.

  • Upshur John A. & Carolyn E. Turner. 1995. Constructing rating scales for second language tests. ELT Journal49(1). 3–12. DOI: 10.1093/elt/49.1.3

  • Winke Paula & Hyojung Lim. 2015. ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing 25. 38–54. DOI: 10.1016/j.asw.2015.05.002

  • Wodak Ruth & Michael Meyer (eds.). 2009. Methods of critical discourse analysis (2nd ed.). Los Angeles CA: Sage.

Search
Journal information
Impact Factor


Cite Score 2018: 0.08

SCImago Journal Rank (SJR) 2018: 0.1
Source Normalized Impact per Paper (SNIP) 2018: 0.095

Metrics
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 407 264 3
PDF Downloads 171 115 2