论文标题
Ernie-search：通过自然蒸馏进行互联网桥接与双重编码器进行桥接编码器，以进行浓密的通道检索
ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval
论文作者
论文摘要
基于预训练的语言模型（PLM）（例如双重编码器）的神经检索器在开放域问答（QA）的任务上实现了有希望的表现。它们的有效性可以通过结合跨架构知识蒸馏来进一步达到最新的最新。但是，大多数现有研究只是直接采用常规蒸馏方法。他们没有考虑教师和学生具有不同结构的特定情况。在本文中，我们提出了一种新型的蒸馏方法，该方法可以显着推动双重编码器的跨架构蒸馏。我们的方法1）引入了一种自然的蒸馏方法，该方法可以有效地将晚期相互作用（即Colbert）蒸馏到Vanilla Dual-nocoder，而2）结合了级联蒸馏过程，以通过交叉编码器老师进一步提高性能。进行了广泛的实验，以验证我们提出的解决方案优于强基础，并在开放域QA基准上建立新的最先进的基准。
Neural retrievers based on pre-trained language models (PLMs), such as dual-encoders, have achieved promising performance on the task of open-domain question answering (QA). Their effectiveness can further reach new state-of-the-arts by incorporating cross-architecture knowledge distillation. However, most of the existing studies just directly apply conventional distillation methods. They fail to consider the particular situation where the teacher and student have different structures. In this paper, we propose a novel distillation method that significantly advances cross-architecture distillation for dual-encoders. Our method 1) introduces a self on-the-fly distillation method that can effectively distill late interaction (i.e., ColBERT) to vanilla dual-encoder, and 2) incorporates a cascade distillation process to further improve the performance with a cross-encoder teacher. Extensive experiments are conducted to validate that our proposed solution outperforms strong baselines and establish a new state-of-the-art on open-domain QA benchmarks.
