论文标题
对变压器和经常性的神经机器翻译方法的精细颗粒人类评估
Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
论文作者
论文摘要
这项研究提出了一种细粒度的人类评估，以在翻译方向英语对英语上的转换器和神经机器翻译（MT）进行比较。为此，我们开发了一个错误的分类法，符合多维质量指标（MQM）框架，该框架是根据该翻译方向的相关现象定制的。然后，我们使用此自定义的错误分类法对WMT2019新闻测试集的子集的最先进的经常性和基于变压器的MT系统的输出进行了错误注释。结果注释表明，与最佳的复发系统相比，最佳的变压器系统导致误差总数减少了31％，并且在22个错误类别中的10个中，误差少得多。我们还注意到，在NMT系统出现之前，评估的两个系统不会为与此翻译方向相关的类别产生任何错误：中文分类器。
This research presents a fine-grained human evaluation to compare the Transformer and recurrent approaches to neural machine translation (MT), on the translation direction English-to-Chinese. To this end, we develop an error taxonomy compliant with the Multidimensional Quality Metrics (MQM) framework that is customised to the relevant phenomena of this translation direction. We then conduct an error annotation using this customised error taxonomy on the output of state-of-the-art recurrent- and Transformer-based MT systems on a subset of WMT2019's news test set. The resulting annotation shows that, compared to the best recurrent system, the best Transformer system results in a 31% reduction of the total number of errors and it produced significantly less errors in 10 out of 22 error categories. We also note that two of the systems evaluated do not produce any error for a category that was relevant for this translation direction prior to the advent of NMT systems: Chinese classifiers.
