论文标题
Sobolev和BESOV空间上深度Relu神经网络的最佳近似率
Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces
论文作者
论文摘要
令$ω= [0,1]^d $为$ \ mathbb {r}^d $中的单位立方体。我们研究了与参数数量有效的问题，具有relu激活功能的深神经网络可以在Sobolev空间中近似函数$ W^s（l_q（ω））$和BESOV Space $ b^s_r（l_q（ω））$，并在$ l_p（ω）中测量错误。当研究神经网络在各种领域（包括科学计算和信号处理）中的应用时，此问题很重要，并且以前仅在$ p = q = q = \ infty $时才解决。我们的贡献是为所有$ 1 \ leq p，q \ leq \ infty $和$ s> 0 $提供完整的解决方案，该解决方案将相应的Sobolev或Besov Space紧凑地嵌入$ L_P $中。关键技术工具是一种新型的位萃取技术，可提供稀疏向量的最佳编码。这使我们能够在$ p> q $的非线性方案中获得尖锐的上限。我们还提供了一种新颖的方法，用于在$ p <\ infty $时根据VC-dimension得出$ L_P $ - APPROXIMATION下限。我们的结果表明，在参数数量方面，非常深的Relu网络明显优于近似的经典方法，但这是以不可编码的参数为代价的。
Let $Ω= [0,1]^d$ be the unit cube in $\mathbb{R}^d$. We study the problem of how efficiently, in terms of the number of parameters, deep neural networks with the ReLU activation function can approximate functions in the Sobolev spaces $W^s(L_q(Ω))$ and Besov spaces $B^s_r(L_q(Ω))$, with error measured in the $L_p(Ω)$ norm. This problem is important when studying the application of neural networks in a variety of fields, including scientific computing and signal processing, and has previously been solved only when $p=q=\infty$. Our contribution is to provide a complete solution for all $1\leq p,q\leq \infty$ and $s > 0$ for which the corresponding Sobolev or Besov space compactly embeds into $L_p$. The key technical tool is a novel bit-extraction technique which gives an optimal encoding of sparse vectors. This enables us to obtain sharp upper bounds in the non-linear regime where $p > q$. We also provide a novel method for deriving $L_p$-approximation lower bounds based upon VC-dimension when $p < \infty$. Our results show that very deep ReLU networks significantly outperform classical methods of approximation in terms of the number of parameters, but that this comes at the cost of parameters which are not encodable.
