论文标题
不确定性决定了模式的充分性和在序列模型中解码的障碍
Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models
论文作者
论文摘要
在许多自然语言处理（NLP）任务中，相同的输入（例如源句子）可以具有多个可能的输出（例如翻译）。为了分析这种歧义（也称为固有的不确定性）如何塑造通过神经序列模型学到的分布，我们通过计算从两个不同的NLP任务中的参考测试集中的参考文献之间的重叠程度来测量句子级别的不确定性：机器翻译（MT）和语法误差校正（GEC）。在句子和任务级别上，内在的不确定性都对搜索的各个方面（例如光束搜索中的归纳偏见和精确搜索的复杂性）都有重大影响。特别是，我们表明，诸如大量梁搜索错误，模式不足以及具有大型大小尺寸的系统性能下降的众所周知的病理适用于具有高水平的模棱两可的任务，例如MT，但不适合GEC等不确定的任务。此外，我们为神经序列模型提出了一种新颖的精确$ n $ best搜索算法，并表明固有的不确定性会影响模型的不确定性，因为该模型倾向于在不确定的任务和句子中过度分散概率质量。
In many natural language processing (NLP) tasks the same input (e.g. source sentence) can have multiple possible outputs (e.g. translations). To analyze how this ambiguity (also known as intrinsic uncertainty) shapes the distribution learned by neural sequence models we measure sentence-level uncertainty by computing the degree of overlap between references in multi-reference test sets from two different NLP tasks: machine translation (MT) and grammatical error correction (GEC). At both the sentence- and the task-level, intrinsic uncertainty has major implications for various aspects of search such as the inductive biases in beam search and the complexity of exact search. In particular, we show that well-known pathologies such as a high number of beam search errors, the inadequacy of the mode, and the drop in system performance with large beam sizes apply to tasks with high level of ambiguity such as MT but not to less uncertain tasks such as GEC. Furthermore, we propose a novel exact $n$-best search algorithm for neural sequence models, and show that intrinsic uncertainty affects model uncertainty as the model tends to overly spread out the probability mass for uncertain tasks and sentences.
