论文标题
联邦加速随机梯度下降
Federated Accelerated Stochastic Gradient Descent
论文作者
论文摘要
我们提出了联合加速的随机梯度下降（FedAc），这是用于分布式优化的联邦平均（FedAvg，也称为局部SGD）的原则加速度。 FedAc是FedAvg的第一个可证明的加速度，可提高各种凸功能的收敛速度和沟通效率。例如，对于强烈的凸和平滑功能，如果使用$ m $工人，则先前的最先进的fedAvg分析可以实现$ m $的线性加速，如果给出了$ m $ $ m $ rounds的同步，而fedac仅需要$ m^{\ frac {\ frac {1}} {3}}} $回合。此外，当目标平滑时，我们证明，对于FedAc，我们证明了更强的保证。我们的技术基于一种基于潜在的扰动迭代分析，对广义加速SGD的新型稳定性分析以及加速和稳定性之间的战略权衡。
We propose Federated Accelerated Stochastic Gradient Descent (FedAc), a principled acceleration of Federated Averaging (FedAvg, also known as Local SGD) for distributed optimization. FedAc is the first provable acceleration of FedAvg that improves convergence speed and communication efficiency on various types of convex functions. For example, for strongly convex and smooth functions, when using $M$ workers, the previous state-of-the-art FedAvg analysis can achieve a linear speedup in $M$ if given $M$ rounds of synchronization, whereas FedAc only requires $M^{\frac{1}{3}}$ rounds. Moreover, we prove stronger guarantees for FedAc when the objectives are third-order smooth. Our technique is based on a potential-based perturbed iterate analysis, a novel stability analysis of generalized accelerated SGD, and a strategic tradeoff between acceleration and stability.
