论文标题
低复杂性LSTM培训和用floatsd8重量表示的推断
Low-Complexity LSTM Training and Inference with FloatSD8 Weight Representation
论文作者
论文摘要
FloatsD技术已被证明在低复杂性卷积神经网络（CNN）训练和推理方面具有出色的性能。在本文中，我们将floatsd应用于复发性神经网络（RNN），特别是长期记忆（LSTM）。除了FloatsD重量表示外，我们还将模型训练中的梯度和激活量化为8位。此外，积累的算术精度和权重的主副本从32位减少到16位。我们证明了提出的培训计划可以从头开始训练多个LSTM模型，同时完全保留了模型的准确性。最后，为了验证所提出的方法在实施方面的优势，我们设计了一个LSTM神经元电路，并表明它可显着降低模具面积和功耗。
The FloatSD technology has been shown to have excellent performance on low-complexity convolutional neural networks (CNNs) training and inference. In this paper, we applied FloatSD to recurrent neural networks (RNNs), specifically long short-term memory (LSTM). In addition to FloatSD weight representation, we quantized the gradients and activations in model training to 8 bits. Moreover, the arithmetic precision for accumulations and the master copy of weights were reduced from 32 bits to 16 bits. We demonstrated that the proposed training scheme can successfully train several LSTM models from scratch, while fully preserving model accuracy. Finally, to verify the proposed method's advantage in implementation, we designed an LSTM neuron circuit and showed that it achieved significantly reduced die area and power consumption.
