论文标题
逆转录：通过蒙版自动编码器进行预训练检索的语言模型
RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder
论文作者
论文摘要
尽管预培训在许多重要的NLP任务中取得了进展，但仍在探索有效的训练前训练策略。在本文中，我们提出了基于蒙版自动编码器（MAE）的新的取回训练范式逆转录。逆瘤由三个关键设计突出显示。 1）新型的MAE工作流程，其中输入句子被用不同的面具污染编码器和解码器。嵌入句子是从编码器的蒙版输入中生成的；然后，根据句子的嵌入和解码器的掩蔽输入通过掩盖语言建模恢复原始句子。 2）非对称模型结构，具有像变压器的全尺度BERT作为编码器，以及一个单层变压器作为解码器。 3）不对称掩蔽比，编码器的比例中等：15〜30％，解码器的侵略性比率：50〜70％。我们的框架很容易实现和凭经验竞争：预先训练的模型会在贝尔和马可女士等广泛的浓密检索基准上显着改善SOTA性能。源代码和预培训模型可在https://github.com/staoxiao/retromae公开获得，以激发更多有趣的研究。
Despite pre-training's progress in many important NLP tasks, it remains to explore effective pre-training strategies for dense retrieval. In this paper, we propose RetroMAE, a new retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE). RetroMAE is highlighted by three critical designs. 1) A novel MAE workflow, where the input sentence is polluted for encoder and decoder with different masks. The sentence embedding is generated from the encoder's masked input; then, the original sentence is recovered based on the sentence embedding and the decoder's masked input via masked language modeling. 2) Asymmetric model structure, with a full-scale BERT like transformer as encoder, and a one-layer transformer as decoder. 3) Asymmetric masking ratios, with a moderate ratio for encoder: 15~30%, and an aggressive ratio for decoder: 50~70%. Our framework is simple to realize and empirically competitive: the pre-trained models dramatically improve the SOTA performances on a wide range of dense retrieval benchmarks, like BEIR and MS MARCO. The source code and pre-trained models are made publicly available at https://github.com/staoxiao/RetroMAE so as to inspire more interesting research.
