Hjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output
We describe Hjerson, a tool for automatic classification of errors in machine translation output. The tool features the detection of five word level error classes: morphological errors, reodering errors, missing words, extra words and lexical errors. As input, the tool requires original full form reference translation(s) and hypothesis along with their corresponding base forms. It is also possible to use additional information on the word level (e.g. pos tags) in order to obtain more details. The tool provides the raw count and the normalised score (error rate) for each error class at the document level and at the sentence level, as well as original reference and hypothesis words labelled with the corresponding error class in text and html formats.
If the inline PDF is not rendering correctly, you can download the PDF file here.
Farrús Mireia Marta R. Costa-jussà Marc Poh Adolfo Hernández and José B. Mariño. Improving a Catalan-Spanish statistical translation system using morphosyntactic knowledge. In Proceedings of the 13th Annual Conference of the European Association for Machine Translation (EAMT 09) pages 52-57 Barcelona Spain May 2009.
Levenshtein Vladimir Iosifovich. Binary Codes Capable of Correcting Deletions Insertions and Reversals. Soviet Physics Doklady 10(8):707-710 February 1966.
Popović Maja and Aljoscha Burchardt. From Human to Automatic Error Classification for Machine Translation Output. In Proceedings of the 15th Annual Conference of the European Association for Machine Translation (EAMT 11) pages 265-272 Leuven Belgium May 2011.
Popović Maja and Hermann Ney. Word Error Rates: Decomposition over POS classes and Applications for Error Analysis. In Proceedings of the 2nd ACL 07 Workshop on Statistical Machine Translation (WMT 07) pages 48-55 Prague Czech Republic June 2007.
Popović Maja and Hermann Ney. Towards Automatic Error Analysis of Machine Translation Output. Computational Linguistics 37(4):xx-xx December (to appear) 2011.
Vilar David Jia Xu Luis Fernando D'Haro and Hermann Ney. Error Analysis of Statistical Machine Translation Output. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 06) pages 697-702 Genoa Italy May 2006.