Bootstrap Masked Visual Modeling via Hard Patches Mining

TL;DR

This paper introduces Hard Patches Mining (HPM), a novel masked visual modeling approach that dynamically identifies and masks challenging patches to improve representation learning in images and videos.

Contribution

It proposes an auxiliary loss predictor and an easy-to-hard mask strategy to enhance masked visual modeling by focusing on hard-to-reconstruct patches, leading to better representations.

Findings

HPM significantly improves performance on image benchmarks.

HPM also enhances video representation learning.

Loss prediction alone benefits the learned representations.

Abstract

Masked visual modeling has attracted much attention due to its promising potential in learning generalizable representations. Typical approaches urge models to predict specific contents of masked tokens, which can be intuitively considered as teaching a student (the model) to solve given problems (predicting masked contents). Under such settings, the performance is highly correlated with mask strategies (the difficulty of provided problems). We argue that it is equally important for the model to stand in the shoes of a teacher to produce challenging problems by itself. Intuitively, patches with high values of reconstruction loss can be regarded as hard samples, and masking those hard patches naturally becomes a demanding reconstruction task. To empower the model as a teacher, we propose Hard Patches Mining (HPM), predicting patch-wise losses and subsequently determining where to mask. Technically, we introduce an auxiliary loss predictor, which is trained with a relative objective to prevent overfitting to exact loss values. Also, to gradually guide the training procedure, we propose an easy-to-hard mask strategy. Empirically, HPM brings significant improvements under both image and video benchmarks. Interestingly, solely incorporating the extra loss prediction objective leads to better representations, verifying the efficacy of determining where is hard to reconstruct. The code is available at https://github.com/Haochen-Wang409/HPM.