论文标题
通过学习反向目标来扩展差异目标传播
Towards Scaling Difference Target Propagation by Learning Backprop Targets
论文作者
论文摘要
生物学上的学习算法的发展对于理解大脑的学习很重要，但是大多数人都无法扩展到现实世界中的任务，从而限制了它们作为真正大脑学习的解释的潜力。因此，重要的是要探索具有强大理论保证的学习算法，并且可以符合复杂任务上的反向传播（BP）的性能。一种这样的算法是差异目标传播（DTP），这是一种具有生物学上的学习算法，该算法最近与Gauss-Newton（GN）优化的密切关系已被建立。但是，这种连接严格保持的条件排除了反馈途径突触权重的层面训练（在生物学上更合理）。此外，DTP重量更新和损耗梯度之间的良好对齐仅是宽松的保证，在非常具体的条件下，培训的体系结构。在本文中，我们提出了一种新颖的反馈体重训练方案，该方案可确保DTP近似BP，并且可以在不牺牲任何理论保证的情况下恢复层的反馈重量训练。我们的理论得到了实验结果的证实，我们报告了DTP在CIFAR-10和Imagenet 32 $ \ times $ 32上取得的最佳性能
The development of biologically-plausible learning algorithms is important for understanding learning in the brain, but most of them fail to scale-up to real-world tasks, limiting their potential as explanations for learning by real brains. As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks. One such algorithm is Difference Target Propagation (DTP), a biologically-plausible learning algorithm whose close relation with Gauss-Newton (GN) optimization has been recently established. However, the conditions under which this connection rigorously holds preclude layer-wise training of the feedback pathway synaptic weights (which is more biologically plausible). Moreover, good alignment between DTP weight updates and loss gradients is only loosely guaranteed and under very specific conditions for the architecture being trained. In this paper, we propose a novel feedback weight training scheme that ensures both that DTP approximates BP and that layer-wise feedback weight training can be restored without sacrificing any theoretical guarantees. Our theory is corroborated by experimental results and we report the best performance ever achieved by DTP on CIFAR-10 and ImageNet 32$\times$32
