The Surprising Effectiveness of Canonical Knowledge Distillation for Semantic Segmentation

TL;DR

This paper demonstrates that simple canonical knowledge distillation methods outperform complex, segmentation-specific approaches when training time is properly accounted for, achieving state-of-the-art results with less complexity.

Contribution

It reveals that canonical KD methods are more effective than recent complex methods for semantic segmentation when training compute is matched, challenging the need for task-specific designs.

Findings

Canonical KD outperforms recent segmentation-specific methods under equal compute.

Feature-based distillation achieves state-of-the-art results on Cityscapes and ADE20K.

A ResNet-18 student reaches 99% of the teacher's performance with only a quarter of parameters.

Abstract

Recent knowledge distillation (KD) methods for semantic segmentation introduce increasingly complex hand-crafted objectives, yet are typically evaluated under fixed iteration schedules. These objectives substantially increase per-iteration cost, meaning equal iteration counts do not correspond to equal training budgets. It is therefore unclear whether reported gains reflect stronger distillation signals or simply greater compute. We show that iteration-based comparisons are misleading: when wall-clock compute is matched, canonical logit- and feature-based KD outperform recent segmentation-specific methods. Under extended training, feature-based distillation achieves state-of-the-art ResNet-18 performance on Cityscapes and ADE20K. A PSPNet ResNet-18 student closely approaches its ResNet-101 teacher despite using only one quarter of the parameters, reaching 99% of the teacher's mIoU on Cityscapes (79.0 vs 79.8) and 92% on ADE20K. Our results challenge the prevailing assumption that KD for segmentation requires task-specific mechanisms and suggest that scaling, rather than complex hand-crafted objectives, should guide future method design.