论文标题
伯语符合生物学：蛋白质语言模型中的注意力
BERTology Meets Biology: Interpreting Attention in Protein Language Models
论文作者
论文摘要
事实证明，变压器体系结构可以学习有用的蛋白质分类和生成任务的表示。但是，这些表示形式在可解释性方面提出了挑战。在这项工作中，我们演示了一组通过注意镜头来分析蛋白质变压器模型的方法。我们表明了注意力：（1）捕获蛋白质的折叠结构，连接在基础序列中相距遥远的氨基酸，但在三维结构中在空间上接近，（（2）靶标结合位点，蛋白质的关键功能成分，（3）集中在逐渐增加的复杂的生物物理性质上，具有增加层的增长层。我们发现这种行为在三个变压器架构（Bert，Albert，XLNET）和两个不同的蛋白质数据集中保持一致。我们还提出了注意力与蛋白质结构之间相互作用的三维可视化。可视化和分析的代码可从https://github.com/salesforce/provis获得。
Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. In this work, we demonstrate a set of methods for analyzing protein Transformer models through the lens of attention. We show that attention: (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth. We find this behavior to be consistent across three Transformer architectures (BERT, ALBERT, XLNet) and two distinct protein datasets. We also present a three-dimensional visualization of the interaction between attention and protein structure. Code for visualization and analysis is available at https://github.com/salesforce/provis.
