论文标题
Bertops：研究拓扑视角下的BERT表示
BERTops: Studying BERT Representations under a Topological Lens
论文作者
论文摘要
提出评分功能，以有效理解，分析和学习大规模变压器模型（例如BERT）的高维隐藏表示的各种属性可能是一项具有挑战性的任务。在这项工作中，我们通过使用持续的同源性（pH）研究BERT隐藏表示的拓扑特征来探索一个新的方向。我们提出了一个名为“持久性评分函数（PSF）”的新颖评分函数，该功能：（i）准确捕获高维隐藏表示形式的同源性，并与广泛的数据集的测试集准确性和胜过现有得分指标的测试集准确性，（ii）捕获了更有趣的“每级”级别的“质量”（II级别的“质量”（II级别）。与基线功能相比，扰动使其成为非常强大的代理，（iv）最后，也可以预测广泛的黑盒和白盒对抗攻击方法的攻击成功率。我们广泛的相关实验表明，PSF对与BERT相关的各种NLP任务的实用性。
Proposing scoring functions to effectively understand, analyze and learn various properties of high dimensional hidden representations of large-scale transformer models like BERT can be a challenging task. In this work, we explore a new direction by studying the topological features of BERT hidden representations using persistent homology (PH). We propose a novel scoring function named "persistence scoring function (PSF)" which: (i) accurately captures the homology of the high-dimensional hidden representations and correlates well with the test set accuracy of a wide range of datasets and outperforms existing scoring metrics, (ii) captures interesting post fine-tuning "per-class" level properties from both qualitative and quantitative viewpoints, (iii) is more stable to perturbations as compared to the baseline functions, which makes it a very robust proxy, and (iv) finally, also serves as a predictor of the attack success rates for a wide category of black-box and white-box adversarial attack methods. Our extensive correlation experiments demonstrate the practical utility of PSF on various NLP tasks relevant to BERT.
