论文标题
噪声与应用于过滤神经网络重量矩阵的信息之间的界限
Boundary between noise and information applied to filtering neural network weight matrices
论文作者
论文摘要
深度神经网络已成功地应用于多种问题，在这些问题中，过度参数产生了部分随机的重量矩阵。权重矩阵单数向量与搬运工 - 托马斯分布的比较表明，在奇异值谱中随机性和学习的信息之间存在边界。受这一发现的启发，我们引入了一种用于噪声滤波的算法，该算法既消除了小的奇异值并降低了较大的奇异值的大小，以抵消噪声和频谱信息部分之间水平排斥的影响。对于在存在标签噪声的情况下训练的网络，我们确实发现，由于噪声过滤，概括性能大大提高。
Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.
