rgbF: An Open Source Tool for n-gram Based Automatic Evaluation of Machine Translation Output

Open access

rgbF: An Open Source Tool for n-gram Based Automatic Evaluation of Machine Translation Output

We describe RGBF, a tool for automatic evaluation of machine translation output based on n-gram precision and recall. The tool calculates the F-score averaged on all n-grams of an arbitrary set of distinct units such as words, morphemes, POS tags, etc. The arithmetic mean is used for n-gram averaging. As input, the tool requires reference translation(s) and hypothesis, both containing the same combination of units. The default output is the document level 4-gram F-score of the desired unit combination. The scores at the sentence level can be obtained on demand, as well as precision and/or recall scores, separate unit scores and separate n-gram scores. In addition, weights can be introduced both for n-grams and for units, as well as the desired n-gram order n.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Avramidis Eleftherios Aljoscha Burchardt Christian Federmann Maja Popović Cindy Tscherwinka and David Vilar. Involving language professionals in the evaluation of machine translation. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC12) Istanbul Turkey May 2012.

  • Callison-Burch Chris Cameron Fordyce Philipp Koehn Christof Monz and Josh Schroeder. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT 2008) pages 70-106 Columbus Ohio June 2008.

  • Callison-Burch Chris Philipp Koehn Christof Monz and Josh Schroeder. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT 2009) pages 1-28 Athens Greece March 2009.

  • Callison-Burch Chris Philipp Koehn Christof Monz Kay Peterson Mark Przybocki and Omar Zaidan. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR (WMT 2010) pages 17-53 Uppsala Sweden July 2010.

  • Callison-Burch Chris Philipp Koehn Christof Monz and Omar Zaidan. Findings of the 2011 Workshop on Statistical Machine Translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT 2011) pages 22-64 Edinburgh Scotland July 2011.

  • Melamed I. Dan Ryan Green and Joseph P. Turian. Precision and Recall of Machine Translation. In Proceedings of the Human Language Technology Conference (HLT-NAACL 03) pages 61-63 Edmonton Canada May/June 2003.

  • Papineni Kishore Salim Roukos Todd Ward and Wie-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 02) pages 311-318 Philadelphia PA July 2002.

  • Popović Maja. Morphemes and POS tags for n-gram based evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT 2011) pages 104-107 Edinburgh Scotland July 2011.

  • Popović Maja and Hermann Ney. Syntax-oriented evaluation measures for machine translation output. In Proceedings of the 4th EACL 09 Workshop on Statistical Machine Translation (WMT 2009) pages 29-32 Athens Greece March 2009.

  • Snover Matthew Bonnie Dorr Richard Schwartz Linnea Micciulla and John Makhoul. A Study of Translation Error Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 06) pages 223-231 Boston MA August 2006.

  • Tiedemann Jorg. News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces. In Recent Advances in Natural Language Processing volume V pages 237-248. John Benjamins Amsterdam Borovets Bulgaria 2009.

Search
Journal information
Cited By
Metrics
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 134 71 5
PDF Downloads 75 40 0