论文标题
交互式知识蒸馏
Interactive Knowledge Distillation
论文作者
论文摘要
知识蒸馏是一个标准的教师学习框架，可在训练有素的大型教师网络的指导下培训轻巧的学生网络。作为一种有效的教学策略，互动教学已被广泛地在学校激励学生，其中教师不仅提供知识，而且还向学生提供了建设性的反馈，以提高他们的学习表现。在这项工作中，我们提出了一种交互式知识蒸馏（IAKD）方案，以利用交互式教学策略进行有效的知识蒸馏。在蒸馏过程中，教师和学生网络之间的相互作用是通过交换操作实现的：用教师网络中的相应块随机替换学生网络中的块。顺便说一句，我们直接涉及教师的强大特征转换能力，以在很大程度上提高学生的表现。具有教师学生网络的典型设置的实验表明，通过我们的IAKD培训的学生网络比在不同的图像分类数据集上接受传统知识蒸馏方法培训的学生网络取得更好的性能。
Knowledge distillation is a standard teacher-student learning framework to train a light-weight student network under the guidance of a well-trained large teacher network. As an effective teaching strategy, interactive teaching has been widely employed at school to motivate students, in which teachers not only provide knowledge but also give constructive feedback to students upon their responses, to improve their learning performance. In this work, we propose an InterActive Knowledge Distillation (IAKD) scheme to leverage the interactive teaching strategy for efficient knowledge distillation. In the distillation process, the interaction between teacher and student networks is implemented by a swapping-in operation: randomly replacing the blocks in the student network with the corresponding blocks in the teacher network. In the way, we directly involve the teacher's powerful feature transformation ability to largely boost the student's performance. Experiments with typical settings of teacher-student networks demonstrate that the student networks trained by our IAKD achieve better performance than those trained by conventional knowledge distillation methods on diverse image classification datasets.
