论文标题
用于培训和修剪卷积神经网络的递归最小二乘
Recursive Least Squares for Training and Pruning Convolutional Neural Networks
论文作者
论文摘要
卷积神经网络（CNN）在许多实际应用中都取得了成功。但是，它们的高计算和存储要求通常会使它们难以在资源受限的设备上部署。为了解决此问题，已经提出了许多针对CNN的修剪算法，但是其中大多数无法将CNN修剪成合理的水平。在本文中，我们提出了一种基于递归最小二乘（RLS）优化的训练和修剪CNN的新型算法。在训练CNN的某些时期后，我们的算法结合了反向输入自相关矩阵和权重矩阵，以评估和修剪不重要的输入通道或节点一层。然后，我们的算法将继续训练修剪的网络，直到修剪网络恢复旧网络的全部性能之前，不会进行下一个修剪。除CNN外，所提出的算法可用于前馈神经网络（FNN）。关于MNIST，CIFAR-10和SVHN数据集的三个实验表明，与其他四种流行的修剪算法相比，我们的算法可以实现更合理的修剪并具有更高的学习效率。
Convolutional neural networks (CNNs) have succeeded in many practical applications. However, their high computation and storage requirements often make them difficult to deploy on resource-constrained devices. In order to tackle this issue, many pruning algorithms have been proposed for CNNs, but most of them can't prune CNNs to a reasonable level. In this paper, we propose a novel algorithm for training and pruning CNNs based on the recursive least squares (RLS) optimization. After training a CNN for some epochs, our algorithm combines inverse input autocorrelation matrices and weight matrices to evaluate and prune unimportant input channels or nodes layer by layer. Then, our algorithm will continue to train the pruned network, and won't do the next pruning until the pruned network recovers the full performance of the old network. Besides for CNNs, the proposed algorithm can be used for feedforward neural networks (FNNs). Three experiments on MNIST, CIFAR-10 and SVHN datasets show that our algorithm can achieve the more reasonable pruning and have higher learning efficiency than other four popular pruning algorithms.
