论文标题
通过功率系列扩展来表征NTK的光谱
Characterizing the Spectrum of the NTK via a Power Series Expansion
论文作者
论文摘要
在网络初始化的轻度条件下，我们得出了无限宽度极限以任意深度进发电率网络的神经切线内核（NTK）的功率系列扩展。我们为该功率系列的系数提供表达式，该系数均取决于激活函数的HERMITE系数以及网络的深度。我们观察到更快的HERMITE系数会导致NTK系数的衰减更快，并探索深度的作用。使用此系列，首先，我们将NTK的有效等级与输入数据克的有效等级联系起来。其次，对于在球体上均匀绘制的数据，我们研究了NTK的特征值，分析了激活函数选择的影响。最后，对于具有足够快的HERMITE系数衰减的通用数据和激活函数，我们在NTK频谱上得出了渐近上限。
Under mild conditions on the network initialization we derive a power series expansion for the Neural Tangent Kernel (NTK) of arbitrarily deep feedforward networks in the infinite width limit. We provide expressions for the coefficients of this power series which depend on both the Hermite coefficients of the activation function as well as the depth of the network. We observe faster decay of the Hermite coefficients leads to faster decay in the NTK coefficients and explore the role of depth. Using this series, first we relate the effective rank of the NTK to the effective rank of the input-data Gram. Second, for data drawn uniformly on the sphere we study the eigenvalues of the NTK, analyzing the impact of the choice of activation function. Finally, for generic data and activation functions with sufficiently fast Hermite coefficient decay, we derive an asymptotic upper bound on the spectrum of the NTK.
