Collaborative Group Learning

TL;DR

This paper introduces Collaborative Group Learning, a novel framework that enhances diversity among student networks through modular routing and subgroup knowledge exchange, leading to improved generalization in image and text tasks.

Contribution

It proposes a flexible modular routing mechanism and subgroup-based knowledge distillation to mitigate homogenization and boost performance in collaborative learning.

Findings

Outperforms state-of-the-art collaborative methods on image and text tasks.

Enhances model generalization without increasing computational costs.

Effectively maintains diversity among student networks.

Abstract

Collaborative learning has successfully applied knowledge transfer to guide a pool of small student networks towards robust local minima. However, previous approaches typically struggle with drastically aggravated student homogenization when the number of students rises. In this paper, we propose Collaborative Group Learning, an efficient framework that aims to diversify the feature representation and conduct an effective regularization. Intuitively, similar to the human group study mechanism, we induce students to learn and exchange different parts of course knowledge as collaborative groups. First, each student is established by randomly routing on a modular neural network, which facilitates flexible knowledge communication between students due to random levels of representation sharing and branching. Second, to resist the student homogenization, students first compose diverse feature sets by exploiting the inductive bias from sub-sets of training data, and then aggregate and distill different complementary knowledge by imitating a random sub-group of students at each time step. Overall, the above mechanisms are beneficial for maximizing the student population to further improve the model generalization without sacrificing computational efficiency. Empirical evaluations on both image and text tasks indicate that our method significantly outperforms various state-of-the-art collaborative approaches whilst enhancing computational efficiency.