Seeing the Whole Picture: Distribution-Guided Data-Free Distillation for Semantic Segmentation

TL;DR

This paper introduces DFSS, a data-free distillation framework for semantic segmentation that leverages Batch Normalization statistics and distribution-guided sampling to improve performance without using original training data.

Contribution

The paper proposes a novel data-free distillation method for semantic segmentation that respects scene continuity and uses BN statistics for better data approximation.

Findings

DFSS outperforms existing data-free methods on standard benchmarks.

The approach achieves state-of-the-art results with less reliance on auxiliary data.

Dynamic sample prioritization improves training efficiency.

Abstract

Semantic segmentation requires a holistic understanding of the physical world, as it assigns semantic labels to spatially continuous and structurally coherent objects rather than to isolated pixels. However, existing data-free knowledge distillation (DFKD) methods-primarily designed for classification-often disregard this continuity, resulting in significant performance degradation when applied directly to segmentation tasks. In this paper, we introduce DFSS, a novel data-free distillation framework tailored for semantic segmentation. Unlike prior approaches that treat pixels independently, DFSS respects the structural and contextual continuity of real-world scenes. Our key insight is to leverage Batch Normalization (BN) statistics from a teacher model to guide Approximate Distribution Sampling (ADS), enabling the selection of data that better reflects the original training distribution-without relying on potentially misleading teacher predictions. Additionally, we propose Weighted Distribution Progressive Distillation (WDPD), which dynamically prioritizes reliable samples that are more closely aligned with the original data distribution early in training and gradually incorporates more challenging cases, mirroring the natural progression of learning in human perception. Extensive experiments on standard benchmarks demonstrate that DFSS consistently outperforms existing data-free distillation methods for semantic segmentation, achieving state-of-the-art results with significantly reduced reliance on auxiliary data.