Heuristic-Free Multi-Teacher Learning

TL;DR

Teacher2Task is a new multi-teacher learning framework that avoids manual heuristics by transforming training into multiple tasks, improving label aggregation and reducing errors across various models and modalities.

Contribution

It introduces teacher-specific input tokens and a reformulated training process to eliminate the need for heuristic-based label aggregation in multi-teacher learning.

Findings

Demonstrates strong empirical results across architectures, modalities, and tasks.

Reduces aggregation errors compared to heuristic-based methods.

Improves label prediction accuracy in multi-teacher settings.

Abstract

We introduce Teacher2Task, a novel framework for multi-teacher learning that eliminates the need for manual aggregation heuristics. Existing multi-teacher methods typically rely on such heuristics to combine predictions from multiple teachers, often resulting in sub-optimal aggregated labels and the propagation of aggregation errors. Teacher2Task addresses these limitations by introducing teacher-specific input tokens and reformulating the training process. Instead of relying on aggregated labels, the framework transforms the training data, consisting of ground truth labels and annotations from N teachers, into N+1 distinct tasks: N auxiliary tasks that predict the labeling styles of the N individual teachers, and one primary task that focuses on the ground truth labels. This approach, drawing upon principles from multiple learning paradigms, demonstrates strong empirical results across a range of architectures, modalities, and tasks.