论文标题
Bitimebert：使用双期信息延长预训练的语言表示
BiTimeBERT: Extending Pre-Trained Language Representations with Bi-Temporal Information
论文作者
论文摘要
时间是文档的重要方面，用于一系列NLP和IR任务。在这项工作中，我们研究了在预训练期间合并时间信息的方法，以进一步提高与时间相关的任务的性能。将使用同步文档集合（例如BookCorpus和Wikipedia）作为培训语料库（例如培训语料库）等常见的预训练的语言模型（例如Bert），我们使用长期跨越的时间新闻文章收集来构建单词表示。我们介绍了Bitimebert，这是一种新颖的语言表示模型，该模型通过两项新的预训练任务培训了新闻文章的时间集合，该任务利用了两个不同的时间信号来构建时间吸引力的语言表示。实验结果表明，Bitimebert始终优于BERT和其他现有的预训练模型，在不同的下游NLP任务和应用程序上，时间很重要（例如，BERT的准确性提高为155 \％\％\％\％\％）。
Time is an important aspect of documents and is used in a range of NLP and IR tasks. In this work, we investigate methods for incorporating temporal information during pre-training to further improve the performance on time-related tasks. Compared with common pre-trained language models like BERT which utilize synchronic document collections (e.g., BookCorpus and Wikipedia) as the training corpora, we use long-span temporal news article collection for building word representations. We introduce BiTimeBERT, a novel language representation model trained on a temporal collection of news articles via two new pre-training tasks, which harnesses two distinct temporal signals to construct time-aware language representations. The experimental results show that BiTimeBERT consistently outperforms BERT and other existing pre-trained models with substantial gains on different downstream NLP tasks and applications for which time is of importance (e.g., the accuracy improvement over BERT is 155\% on the event time estimation task).
