Distilling Vision Transformers for Distortion-Robust Representation Learning

TL;DR

This paper introduces a knowledge distillation method where a Vision Transformer learns distortion-robust representations by distilling from a clean-image teacher to a distorted-image student, improving classification under various distortions.

Contribution

It proposes an asymmetric multi-level distillation framework leveraging pretrained Vision Transformers to learn distortion-robust visual representations without clean data.

Findings

Student models outperform existing methods on distorted image classification.

Multi-level distillation aligns global, patch, and attention features effectively.

The approach works across multiple datasets and distortion types.

Abstract

Self-supervised learning has achieved remarkable success in learning visual representations from clean data, yet remains challenging when clean observations are sparse or not available at all. In this paper, we demonstrate that pretrained vision models can be leveraged to learn distortion-robust representations, which can then be effectively applied to downstream tasks operating on distorted observations. In particular, we propose an asymmetric knowledge distillation framework in which both teacher and student are initialized from the same pretrained Vision Transformer but receive different views of each image: the teacher processes clean images, while the student sees their distorted versions. We introduce multi-level distillation that aligns global embeddings, patch-level features, and attention maps and show that the student is able to approximate clean-image representations despite never directly accessing clean data. We evaluate our approach on image classification tasks across several datasets and under various distortions, consistently outperforming existing alternatives for the same amount of human supervision.