论文标题
[重新]通过知识审查提炼知识
[Re] Distilling Knowledge via Knowledge Review
论文作者
论文摘要
这项工作旨在重现实验结果，并分析CVPR '21论文“通过知识评论蒸馏知识通过知识回顾”中引入的知识蒸馏的鲁棒性。以前的知识蒸馏作品仅研究了学生和老师相同级别之间的连接路径，尚未考虑跨层次的连接路径。 Chen等。提出一个新的残留学习框架，以使用多个教师层来培训单个学生层。他们还设计了一种新颖的融合模块，以跨越层次的凝结特征图和损失功能，以比较在不同级别上存储的特征信息以提高性能。在这项工作中，我们始终验证原始论文报告的学生模型的测试准确性的提高，并研究通过进行消融研究和新实验引入的新型模块的有效性。
This effort aims to reproduce the results of experiments and analyze the robustness of the review framework for knowledge distillation introduced in the CVPR '21 paper 'Distilling Knowledge via Knowledge Review' by Chen et al. Previous works in knowledge distillation only studied connections paths between the same levels of the student and the teacher, and cross-level connection paths had not been considered. Chen et al. propose a new residual learning framework to train a single student layer using multiple teacher layers. They also design a novel fusion module to condense feature maps across levels and a loss function to compare feature information stored across different levels to improve performance. In this work, we consistently verify the improvements in test accuracy across student models as reported in the original paper and study the effectiveness of the novel modules introduced by conducting ablation studies and new experiments.
