Distilling Invariant Representations with Dual Augmentation

TL;DR

This paper introduces a dual augmentation strategy in knowledge distillation to enhance invariant feature learning, leading to more robust and transferable representations in student models.

Contribution

It proposes a novel dual augmentation method that improves invariant representation learning during knowledge distillation, extending causal interpretation approaches.

Findings

Achieves competitive results on CIFAR-100

Enhances robustness of student models across data variations

Complementary to existing invariant causal distillation methods

Abstract

Knowledge distillation (KD) has been widely used to transfer knowledge from large, accurate models (teachers) to smaller, efficient ones (students). Recent methods have explored enforcing consistency by incorporating causal interpretations to distill invariant representations. In this work, we extend this line of research by introducing a dual augmentation strategy to promote invariant feature learning in both teacher and student models. Our approach leverages different augmentations applied to both models during distillation, pushing the student to capture robust, transferable features. This dual augmentation strategy complements invariant causal distillation by ensuring that the learned representations remain stable across a wider range of data variations and transformations. Extensive experiments on CIFAR-100 demonstrate the effectiveness of this approach, achieving competitive results in same-architecture KD.