论文标题
过度拟合：插值可以证明排除不变性
Malign Overfitting: Interpolation Can Provably Preclude Invariance
论文作者
论文摘要
学识渊博的分类器通常应具有某些不变属性，以鼓励公平，稳健性或分布概括。然而，最近的多项著作在经验上表明，常见的不变性正规化器在过度参数化方案中无效，其中分类器完全拟合（即插值）训练数据。这表明“良性过度拟合”的现象在插值尽管插值良好地概括了，但可能不会有利地扩展到需要鲁棒性或公平性的设置。 在这项工作中，我们为这些观察提供了理论上的理由。我们证明，即使在最简单的设置中，任何插值学习规则（任意少量）也将无法满足这些不变性属性。然后，我们提出和分析一种算法，在同一环境中 - 成功地学习了一个不变的非交流分类器。我们验证了对模拟数据和水鸟数据集的理论观察结果。
Learned classifiers should often possess certain invariance properties meant to encourage fairness, robustness, or out-of-distribution generalization. However, multiple recent works empirically demonstrate that common invariance-inducing regularizers are ineffective in the over-parameterized regime, in which classifiers perfectly fit (i.e. interpolate) the training data. This suggests that the phenomenon of "benign overfitting", in which models generalize well despite interpolating, might not favorably extend to settings in which robustness or fairness are desirable. In this work we provide a theoretical justification for these observations. We prove that -- even in the simplest of settings -- any interpolating learning rule (with arbitrarily small margin) will not satisfy these invariance properties. We then propose and analyze an algorithm that -- in the same setting -- successfully learns a non-interpolating classifier that is provably invariant. We validate our theoretical observations on simulated data and the Waterbirds dataset.
