论文标题
ADACAP：馈送前向神经网络的自适应能力控制
AdaCap: Adaptive Capacity control for Feed-Forward Neural Networks
论文作者
论文摘要
ML模型的容量是指该模型可以近似的功能范围。它影响了模型可以学习的模式的复杂性，但也影响了记忆，即模型适合任意标签的能力。我们提出了自适应能力（ADACAP），这是一种用于前馈神经网络（FFNN）的培训计划。 ADACAP优化了FFNN的能力，因此它可以捕获手头问题的高级抽象表示，而无需记住培训数据集。 ADACAP是两种新成分的组合，即正规化的混乱标签（MLR）损失和Tikhonov操作员培训方案。 MLR损失利用随机生成的标签来量化模型的记忆倾向。我们证明，MLR损失是用于样本外泛化性能的准确样本估计量，并且可以使用信噪比的条件来执行超参数优化。 Tikhonov操作员培训方案以自适应，可区分和数据依赖性方式调节FFNN的能力。我们评估ADACAP在DNN通常容易记忆，小表格数据集并根据流行的机器学习方法进行基准性能的环境中评估ADACAP的有效性。
The capacity of a ML model refers to the range of functions this model can approximate. It impacts both the complexity of the patterns a model can learn but also memorization, the ability of a model to fit arbitrary labels. We propose Adaptive Capacity (AdaCap), a training scheme for Feed-Forward Neural Networks (FFNN). AdaCap optimizes the capacity of FFNN so it can capture the high-level abstract representations underlying the problem at hand without memorizing the training dataset. AdaCap is the combination of two novel ingredients, the Muddling labels for Regularization (MLR) loss and the Tikhonov operator training scheme. The MLR loss leverages randomly generated labels to quantify the propensity of a model to memorize. We prove that the MLR loss is an accurate in-sample estimator for out-of-sample generalization performance and that it can be used to perform Hyper-Parameter Optimization provided a Signal-to-Noise Ratio condition is met. The Tikhonov operator training scheme modulates the capacity of a FFNN in an adaptive, differentiable and data-dependent manner. We assess the effectiveness of AdaCap in a setting where DNN are typically prone to memorization, small tabular datasets, and benchmark its performance against popular machine learning methods.
