Rethinking Random Masking in Self-Distillation on ViT

TL;DR

This paper investigates the impact of selective random masking in self-distillation with ViTs, demonstrating that masking only the student's global view improves robustness and attention quality, leading to better downstream results.

Contribution

It introduces an asymmetric masking strategy in DINO, masking only the student's global view to enhance robustness without losing critical information.

Findings

Masking only the student's global view improves robustness.

The approach yields more fine-grained attention maps.

Enhanced downstream performance on mini-ImageNet.

Abstract

Vision Transformers (ViTs) have demonstrated remarkable performance across a wide range of vision tasks. In particular, self-distillation frameworks such as DINO have contributed significantly to these advances. Within such frameworks, random masking is often utilized to improve training efficiency and introduce regularization. However, recent studies have raised concerns that indiscriminate random masking may inadvertently eliminate critical semantic information, motivating the development of more informed masking strategies. In this study, we explore the role of random masking in the self-distillation setting, focusing on the DINO framework. Specifically, we apply random masking exclusively to the student's global view, while preserving the student's local views and the teacher's global view in their original, unmasked forms. This design leverages DINO's multi-view augmentation scheme to retain clean supervision while inducing robustness through masked inputs. We evaluate our approach using DINO-Tiny on the mini-ImageNet dataset and show that random masking under this asymmetric setup yields more robust and fine-grained attention maps, ultimately enhancing downstream performance.