Open access

Abstract

In the paper, we present EVALD applications (Evaluator of Discourse) for automated essay scoring. EVALD is the first tool of this type for Czech. It evaluates texts written by both native and non-native speakers of Czech. We describe first the history and the present in the automatic essay scoring, which is illustrated by examples of systems for other languages, mainly for English. Then we focus on the methodology of creating the EVALD applications and describe datasets used for testing as well as supervised training that EVALD builds on. Furthermore, we analyze in detail a sample of newly acquired language data – texts written by non-native speakers reaching the threshold level of the Czech language acquisition required e.g. for the permanent residence in the Czech Republic – and we focus on linguistic differences between the available text levels. We present the feature set used by EVALD and – based on the analysis – we extend it with new spelling features. Finally, we evaluate the overall performance of various variants of EVALD and provide the analysis of collected results.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Ajay Helen B. P. I. Tillett and Ellis B. Page. Analysis of Essays by Computer (AEC-II). Final Report to the National Center for Educational Research and Development U.S. Department of Health Education and Welfare (Project No. 80101) page 231 1973.

  • Al-Jouie Maram F. and Aqil M. Azmi. Automated Evaluation of School Children Essays in Arabic. Procedia Computer Science: Arabic Computational Linguistics 117:19 – 22 2017.

  • Boyd Adriane Jirka Hana Lionel Nicolas Detmar Meurers Katrin Wisniewski Andrea Abel Karin Schöne Barbora Štindlová and Chiara Vettori. The MERLIN corpus: Learner language and the CEFR. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014) pages 1281–1288 Reykjavík Iceland 2014. European Language Resources Association.

  • Broda Bartosz Bartłomiej Nitoń Włodzimierz Gruszczyński and Maciej Ogrodniczuk. Measuring Readability of Polish Texts: Baseline Experiments. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14) pages 573–580 Reykjavik Iceland 2014. European Languages Resources Association (ELRA).

  • Burstein Jill Karen Kukich Susanne Wolff Chi Lu and Martin Chodorow. Computer analysis of essays. 1998.

  • Castro-Castro Daniel Rocío Lannes-Losada Montse Maritxalar Ianire Niebla Celia Pérez-Marqués Nancy C. Álamo-Suárez and Aurora Pons-Porrata. A Multilingual Application for Automated Essay Scoring. In Advances in Artificial Intelligence – IBERAMIA 2008 pages 243–251 Berlin Heidelberg 2008. Springer Berlin Heidelberg.

  • Foltz Peter W. Darrell Laham and Thomas K. Landauer. The Intelligent Essay Assessor: Applications to Educational Technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning 1(2):939–944 1999.

  • Hnátková Milena Michal Křen Pavel Procházka and Hana Skoumalová. The SYN-series Corpora of Written Czech. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14) pages 160–164 Reykjavik Iceland 2014. European Language Resources Association (ELRA).

  • Ishioka Tsunenori and Masayuki Kameda. Automated Japanese Essay Scoring System Based on Articles Written by Experts. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics pages 233–240 Stroudsburg PA USA 2006. Association for Computational Linguistics.

  • Koehn Philipp. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing pages 388–395 Barcelona Spain 2004. Association for Computational Linguistics.

  • Larkey Leah S. Automatic Essay Grading Using Text Categorization Techniques. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval pages 90–95 New York NY USA 1998. Association for Computing Machinery.

  • Lemaire Benoit and Philippe Dessus. A System to Assess the Semantic Content of Student Essays. Journal of Educational Computing Research 24(3):305–320 2001.

  • Novák Michal Kateřina Rysová Magdaléna Rysová and Jiří Mírovský. Incorporating Coreference to Automatic Evaluation of Coherence in Essays. In Statistical Language and Speech Processing number 10583 in Lecture Notes in Computer Science pages 58–69 Cham Switzerland 2017. Springer International Publishing.

  • Novák Michal Jiří Mírovský Kateřina Rysová and Magdaléna Rysová. Topic–Focus Articulation: A Third Pillar of Automatic Evaluation of Text Coherence. In Advances in Computational Intelligence number 11289 in Lecture Notes in Computer Science pages 96–108 Cham Switzerland 2018. Springer International Publishing.

  • Novák Michal Jiří Mírovský Kateřina Rysová and Magdaléna Rysová. Exploiting Large Unlabeled Data in Automatic Evaluation of Coherence in Czech. In Text Speech and Dialogue number 11697 in Lecture Notes in Computer Science pages 197–210 Cham Switzerland 2019. Springer International Publishing.

  • Page Ellis B. The Imminence of… Grading Essays by Computer. Phi Delta Kappan 47(5):238–243 1966.

  • Page Ellis B. The Use of the Computer in Analyzing Student Essays. International Review of Education 14(2):210–225 1968.

  • Page Ellis B. and Nancy S. Petersen. The Computer Moves into Essay Grading: Updating the Ancient Test. Phi Delta Kappan 76(7):561–565 1995.

  • Pedregosa Fabian Gaël Varoquaux Alexandre Gramfort Vincent Michel Bertrand Thirion Olivier Grisel Mathieu Blondel Peter Prettenhofer Ron Weiss Vincent Dubourg Jake Vanderplas Alexandre Passos David Cournapeau Matthieu Brucher Matthieu Perrot and Édouard Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12:2825–2830 2011.

  • Popel Martin and Zdeněk Žabokrtský. TectoMT: Modular NLP Framework. In Proceedings of the 7th International Conference on Advances in Natural Language Processing pages 293–304 Berlin Heidelberg 2010. Springer-Verlag.

  • Rysová Kateřina Magdaléna Rysová and Jiří Mírovský. Automatic evaluation of surface coherence in L2 texts in Czech. In Proceedings of the 28th Conference on Computational Linguistics and Speech Processing ROCLING XXVIII (2016) pages 214–228 Taipei Taiwan 2016. National Cheng Kung University The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). ISBN 978-957-30792-9-3.

  • Rysová Kateřina Magdaléna Rysová Jiří Mírovský and Michal Novák. Introducing EVALD – Software Applications for Automatic Evaluation of Discourse in Czech. In Proceedings of the International Conference Recent Advances in Natural Language Processing pages 634–641 Varna Bulgaria 2017. INCOMA Ltd.

  • Šebesta Karel Zuzanna Bedřichová Kateřina Šormová et al. AKCES 5 (CzeSL-SGT) 2014. data/software LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics Charles University Prague Czech Republic.

  • Šebesta Karel Hana Goláňová Jana Letafková et al. AKCES 1 2016. data/software LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics Charles University Prague Czech Republic.

  • Wild Fridolin Christina Stahl Gerald Stermsek Yoseba Penya and Gustaf Neumann. Factors Influencing Effectiveness in Automated Essay Scoring with LSA. In Proceedings of the Conference on Artificial Intelligence in Education: Supporting Learning Through Intelligent and Socially Informed Technology pages 947–949 Amsterdam The Netherlands 2005. IOS Press.

  • Zupanc Kaja and Zoran Bosnić. Advances in the Field of Automated Essay Evaluation. Informatica 4(39):383–396 2015.

Search
Journal information
Metrics
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 18 18 18
PDF Downloads 10 10 10