论文标题
随着时间的推移，对具有时间截断的局部反向传播的尖峰神经网络的有效培训
Efficient Training of Spiking Neural Networks with Temporally-Truncated Local Backpropagation through Time
论文作者
论文摘要
直接训练尖峰神经网络（SNN）由于复杂的神经动力学和触发功能中的内在非差异性，因此仍然具有挑战性。提议训练SNN的众所周知的反向传播（BPTT）算法遭受了巨大的记忆足迹，并禁止向后和更新解锁，从而无法利用本地监督的训练方法的潜力。这项工作为SNN提出了一种有效，直接的训练算法，该算法将本地监督的训练方法与时间截断的BPTT算法相结合。所提出的算法探索了BPTT中的时间和空间位置，并有助于大大降低计算成本，包括GPU存储器利用率，主内存访问和算术操作。我们彻底探讨了有关时间截断长度和本地培训块大小的设计空间，并基于其对运行不同类型任务的不同网络的分类准确性的影响。结果表明，时间截断对基于帧的数据集进行分类的准确性具有负面影响，但会提高动态视觉传感器（DVS）记录的数据集的准确性。尽管信息丢失了，但本地培训仍能够减轻过度拟合。时间截断和局部训练的综合效果会导致准确性下降的放缓，甚至可以提高准确性。此外，训练深SNNS模型（例如Alexnet分类CIFAR10-DVS数据集）的准确性提高了7.26％，GPU内存降低了89.94％，记忆访问降低了10.79％，与标准的端端端端BPTT相比，MAC的降低了10.79％，减少了99.64％。
Directly training spiking neural networks (SNNs) has remained challenging due to complex neural dynamics and intrinsic non-differentiability in firing functions. The well-known backpropagation through time (BPTT) algorithm proposed to train SNNs suffers from large memory footprint and prohibits backward and update unlocking, making it impossible to exploit the potential of locally-supervised training methods. This work proposes an efficient and direct training algorithm for SNNs that integrates a locally-supervised training method with a temporally-truncated BPTT algorithm. The proposed algorithm explores both temporal and spatial locality in BPTT and contributes to significant reduction in computational cost including GPU memory utilization, main memory access and arithmetic operations. We thoroughly explore the design space concerning temporal truncation length and local training block size and benchmark their impact on classification accuracy of different networks running different types of tasks. The results reveal that temporal truncation has a negative effect on the accuracy of classifying frame-based datasets, but leads to improvement in accuracy on dynamic-vision-sensor (DVS) recorded datasets. In spite of resulting information loss, local training is capable of alleviating overfitting. The combined effect of temporal truncation and local training can lead to the slowdown of accuracy drop and even improvement in accuracy. In addition, training deep SNNs models such as AlexNet classifying CIFAR10-DVS dataset leads to 7.26% increase in accuracy, 89.94% reduction in GPU memory, 10.79% reduction in memory access, and 99.64% reduction in MAC operations compared to the standard end-to-end BPTT.
