论文标题
低位位神经网络的组合优化
Combinatorial optimization for low bit-width neural networks
论文作者
论文摘要
低位宽度神经网络已被广泛探索以在边缘设备上部署，以减少计算资源。现有的方法集中在两阶段的火车和压缩设置中基于梯度的优化，或作为在训练过程中量化梯度的组合优化。此类方案在训练阶段需要高性能硬件，并且通常会在量化的权重以外存储等效数量的完整重量。在本文中，我们探讨了与二进制重量最小化的风险最小化问题的直接组合优化方法，这可以使其等效于在某些条件下的非单调性下管最大化。我们对具有单个和多层神经网络的病例采用了近似算法。对于线性模型，它具有$ \ Mathcal {o}（nd）$ time复杂性，其中$ n $是样本大小，$ d $是数据维度。我们表明，贪婪的坐标下降和这种新颖的方法的结合可以在二进制分类任务上获得竞争精度。
Low-bit width neural networks have been extensively explored for deployment on edge devices to reduce computational resources. Existing approaches have focused on gradient-based optimization in a two-stage train-and-compress setting or as a combined optimization where gradients are quantized during training. Such schemes require high-performance hardware during the training phase and usually store an equivalent number of full-precision weights apart from the quantized weights. In this paper, we explore methods of direct combinatorial optimization in the problem of risk minimization with binary weights, which can be made equivalent to a non-monotone submodular maximization under certain conditions. We employ an approximation algorithm for the cases with single and multilayer neural networks. For linear models, it has $\mathcal{O}(nd)$ time complexity where $n$ is the sample size and $d$ is the data dimension. We show that a combination of greedy coordinate descent and this novel approach can attain competitive accuracy on binary classification tasks.
