论文标题
差异私有模型压缩
Differentially Private Model Compression
论文作者
论文摘要
最近的论文表明，可以在私人数据上微调大型的预训练的语言模型（LLM），例如BERT，GPT-2可以实现与许多下游自然语言处理（NLP）任务相当的性能，同时保证差异隐私。但是，这些模型的推论成本（由数亿个参数组成）可能很大。因此，通常在实践中，LLM在将其部署到特定应用程序中之前被压缩。在本文中，我们启动了差异私有模型压缩的研究，并提出了达到50％稀疏水平的框架，同时保持了几乎完全的性能。我们使用BERT模型在标准胶水基准上展示了这些想法，并为未来的研究设定了基准。
Recent papers have shown that large pre-trained language models (LLMs) such as BERT, GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private models for many downstream Natural Language Processing (NLP) tasks while simultaneously guaranteeing differential privacy. The inference cost of these models -- which consist of hundreds of millions of parameters -- however, can be prohibitively large. Hence, often in practice, LLMs are compressed before they are deployed in specific applications. In this paper, we initiate the study of differentially private model compression and propose frameworks for achieving 50% sparsity levels while maintaining nearly full performance. We demonstrate these ideas on standard GLUE benchmarks using BERT models, setting benchmarks for future research on this topic.
