论文标题
无限深神经网络的变异推断
Variational Inference for Infinitely Deep Neural Networks
论文作者
论文摘要
我们介绍了无界深度神经网络（UDN），这是一个无限深的概率模型，可将其复杂性适应训练数据。 UDN包含一个无限的隐藏层序列，并在截断L上放置了无限的先验，该层产生了其数据。给定观察数据集，后UDN提供了无限神经网络参数及其截断的条件分布。我们开发了一种新型的变异推理算法来近似此后部，优化了神经网络权重和截断深度L的分布，而没有任何上限。到此，该末端具有特殊的结构：它具有特殊的结构：IT模型的神经网络重量是任意深度的神经网络权重，并且它可以动态地造成自由变异参数的分布，是其分布的分布。 （与启发式搜索的方法不同，该算法仅通过基于梯度的优化来探讨截断的空间。）我们在实际和合成数据上研究了UDN。我们发现UDN将其后深度适应数据集的复杂性。它的表现优于类似计算复杂性的标准神经网络；它的表现优于无限深度神经网络的其他方法。
We introduce the unbounded depth neural network (UDN), an infinitely deep probabilistic model that adapts its complexity to the training data. The UDN contains an infinite sequence of hidden layers and places an unbounded prior on a truncation L, the layer from which it produces its data. Given a dataset of observations, the posterior UDN provides a conditional distribution of both the parameters of the infinite neural network and its truncation. We develop a novel variational inference algorithm to approximate this posterior, optimizing a distribution of the neural network weights and of the truncation depth L, and without any upper limit on L. To this end, the variational family has a special structure: it models neural network weights of arbitrary depth, and it dynamically creates or removes free variational parameters as its distribution of the truncation is optimized. (Unlike heuristic approaches to model search, it is solely through gradient-based optimization that this algorithm explores the space of truncations.) We study the UDN on real and synthetic data. We find that the UDN adapts its posterior depth to the dataset complexity; it outperforms standard neural networks of similar computational complexity; and it outperforms other approaches to infinite-depth neural networks.
