论文标题
通过生成对抗网络通过模型蒸馏进行私人知识转移
Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks
论文作者
论文摘要
深度学习应用程序的部署必须在使用私人和敏感数据进行培训时解决日益增长的隐私问题。传统的深度学习模型容易受到隐私攻击，可以从模型参数或访问目标模型中恢复个人的敏感信息。最近，已提议提供可证明的隐私保证的差异隐私以保护培训数据来培训神经网络。但是，许多方法倾向于为模型发布提供最坏的案例隐私保证，不可避免地会损害训练有素的模型的准确性。在本文中，我们提出了一种新颖的私人知识转移策略，在该策略中，接受敏感数据培训的私人老师无法公开访问，但教导学生公开发布。特别是，提出了一个三人（教师 - 歧视者）学习框架，以实现效用和隐私之间的权衡，在该效用和隐私之间，学生从教师那里获取蒸馏知识，并接受了歧视者的培训，以产生与老师相似的成果。然后，我们将差异性隐私保护机制集成到学习过程中，从而使培训具有严格的隐私预算。该框架最终允许仅使用未标记的公共数据和很少的时代培训学生，因此可以防止敏感培训数据的暴露，同时确保具有适度的隐私预算的模型实用程序。关于MNIST，SVHN和CIFAR-10数据集的实验表明，我们的学生获得了0.89％，2.29％，5.16％的准确性损失，分别具有（1.93，10^-5）的隐私范围（（5.02，10^-6），（8.81，10^-6），（8.81，10^-6）。与现有作品\ Cite {PaperNot2016Semi，Wang2019-Private}相比，提议的工作可以提高5-82％的精度损失。
The deployment of deep learning applications has to address the growing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information of individuals from either model parameters or accesses to the target model. Recently, differential privacy that offers provable privacy guarantees has been proposed to train neural networks in a privacy-preserving manner to protect training data. However, many approaches tend to provide the worst case privacy guarantees for model publishing, inevitably impairing the accuracy of the trained models. In this paper, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but teaches a student to be publicly released. In particular, a three-player (teacher-student-discriminator) learning framework is proposed to achieve trade-off between utility and privacy, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. We then integrate a differential privacy protection mechanism into the learning procedure, which enables a rigorous privacy budget for the training. The framework eventually allows student to be trained with only unlabelled public data and very few epochs, and hence prevents the exposure of sensitive training data, while ensuring model utility with a modest privacy budget. The experiments on MNIST, SVHN and CIFAR-10 datasets show that our students obtain the accuracy losses w.r.t teachers of 0.89%, 2.29%, 5.16%, respectively with the privacy bounds of (1.93, 10^-5), (5.02, 10^-6), (8.81, 10^-6). When compared with the existing works \cite{papernot2016semi,wang2019private}, the proposed work can achieve 5-82% accuracy loss improvement.
