ORC: Network Group-based Knowledge Distillation using Online Role Change
Junyong Choi, Hyeon Cho, Seokhwa Cheung, Wonjun Hwang

TL;DR
This paper introduces ORC, a network group-based knowledge distillation method that dynamically switches networks between teacher and student roles to improve learning effectiveness, validated on multiple datasets and architectures.
Contribution
It proposes an online role change strategy for network groups, enhancing knowledge transfer by selecting and promoting networks between teacher and student roles during training.
Findings
Achieves superior performance on CIFAR-10, CIFAR-100, and ImageNet.
Demonstrates effectiveness across various backbone architectures.
Improves knowledge distillation by dynamic role assignment.
Abstract
In knowledge distillation, since a single, omnipotent teacher network cannot solve all problems, multiple teacher-based knowledge distillations have been studied recently. However, sometimes their improvements are not as good as expected because some immature teachers may transfer the false knowledge to the student. In this paper, to overcome this limitation and take the efficacy of the multiple networks, we divide the multiple networks into teacher and student groups, respectively. That is, the student group is a set of immature networks that require learning the teacher's knowledge, while the teacher group consists of the selected networks that are capable of teaching successfully. We propose our online role change strategy where the top-ranked networks in the student group are able to promote to the teacher group at every iteration. After training the teacher group using the error…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
ORC: Network Group-based Knowledge Distillation using Online Role Change· youtube
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Advanced Image and Video Retrieval Techniques
