Adopting Weir’s (2005) socio-cognitive validation framework, the present paper focuses on the syntactic and lexical complexity of listening comprehension subtests in three B2-level examinations: The City Guilds international examination in English, The First Certificate in English, and the General Matura in English. By analysing and interpreting the results obtained from different automated tools, the research aims to determine to what extent the three subtests are comparable. The results of the study suggest the unreliability of the Common European Framework of Reference for Languages (CEFR) as a sole mechanism for test comparisons.
Alderson, Charles J. 2000. Assessing Reading. Cambridge: Cambridge University Press.
Alderson, J. Charles. 2006. “Analysing tests of reading and listening in relation to the common European framework of reference: The experience of the Dutch CEFR Construct Project.” Language Assessment Quarterly 3(1):3-30.
Alderson, J. Charles, Neus Figueras, Henk Kuijper, Guenter Nold, Sauli Takala and Claire Tardieu. 2004. “The development of specifications for item development and classification within The Common European Framework of Reference for Languages: Learning, Teaching, Assessment: Reading and Listening: Final report of The Dutch CEF Construct Project. Working paper” in Lancaster University Publications & Outputs [Online]. Available: http://eprints.lancs.ac.uk/44/1/final_report.pdf [Accessed 2015, June 11].
American Psychological Association, American Educational Research Association and National Council on Measurement in Education. 1954. Technical recommendations for psychological tests and diagnostic techniques. Washington: The Association.
Bloomfield, Amber. N., Sarah C. Wayland, Allison Blodgett and Jared Linck. 2011. “Factors Related to Passage Length: Implications for Second Language Listening Comprehension” in Expanding the Space of Cognitive Science, Proceedings of the 33rd Annual Meeting of the Cognitive Science Society. Laura Carlson, Christoph Hoelscher and Thomas. F. Shipley (Eds.). Austin: Cognitive Science Society, pp. 2317-2322.
Buck, Gary. 2001. Assessing Listening. Cambridge: Cambridge University Press.
Cobb, T. n.d. “Web Vocabprofile, an adaptation of Heatley, Nation & Coxhead’s (2002) Range.” [Online]. Available: http://www.lextutor.ca/vp/ [Accessed 2015, March 12].
Halliday, M. A. K. and Ruqaiya Hasan. 1976. Cohesion in English. London: Longman.
Ilc, Gašper and Andrej Stopar. 2015. “Validating the Slovenian national alignment to CEFR: The case of the B2 reading comprehension examination in English.” Language Testing 32(4):443-462.
Jones, Neil. 2009. “A comparative approach to constructing a multilingual proficiency framework: constraining the role of standard-setting” in Linking to the CEFR levels: Research perspectives. [Online]. Neus Figueras and José Noijons (Eds.). Arnhem: Cito, EALTA, pp. 33-43. Available: http://www.coe.int/t/dg4/linguistic/Proceedings_CITO_EN.pdf [Accessed 2015, March 14].
Kobeleva, Polina P. 2012. “Second Language Listening and Unfamiliar Proper Names: Comprehension Barrier?” RELC Journal 43(1):83-98.
Matthews, Joshua and Junyu Cheng. 2015. “Recognition of high frequency words from speech as a predictor of L2 listening comprehension.” System 52:1-13.
Messick, Samuel. 1995. “Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning.” American Psychologist 50:741-749.
McNamara, Danielle S., Arthur C. Graesser, Philip M. McCarthy and Zhiqiang Cai. 2014. Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press.
McNamara, Danielle S., Max M. Louwerse, Zhiqiang Cai and Arthur C. Graesser. 2005. Coh-Metrix version 1.4. [Online]. Available: http://cohmetrix.memphis.edu [Accessed April, 2015].
Rost, Michael. 1990. Listening in Language Learning. London: Longman.
Rost, Michael. 2006. “Areas of research that influence L2 listening instruction” in Current Trends in the Development and Teaching of the Four Language Skills. Esther Uso-Juan and Alicia Martinez-Flor (Eds.). New York: Mouton de Gruyter, pp. 47-74.
Shiotsu, Toshihiko and Cyril J. Weir. 2007. “The relative significance of syntactic knowledge and vocabulary breadth in the prediction of reading comprehension test performance.” Language Testing 24(1):99-128.
Shohamy, Elana and Ofra Inbar. 1991. “Validation of listening comprehension tests: the effect of text and question type.” Language Testing 8(1):23-40.
Taylor, Lynda. 2004. “Issues of test comparability.” Research Notes 15:2-5.
Weir, Cyril J. 2005. Language testing and validation. An Evidence-Based Approach. Houndgrave: Palgrave Macmillan.