Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning

TL;DR

This paper investigates how the choice of reconstruction target in masked image modeling affects cross-domain few-shot learning, proposing a new method that balances domain-agnostic features and global structure for improved transferability.

Contribution

It reveals the importance of reconstruction targets in CDFSL and introduces DAMIM, a novel approach with feature aggregation and lightweight decoding to enhance cross-domain transfer.

Findings

Reconstruction target impacts transferability in CDFSL.

Reconstructing high-level features offers limited benefits.

DAMIM achieves state-of-the-art results on four datasets.

Abstract

Cross-Domain Few-Shot Learning (CDFSL) requires the model to transfer knowledge from the data-abundant source domain to data-scarce target domains for fast adaptation, where the large domain gap makes CDFSL a challenging problem. Masked Autoencoder (MAE) excels in effectively using unlabeled data and learning image's global structures, enhancing model generalization and robustness. However, in the CDFSL task with significant domain shifts, we find MAE even shows lower performance than the baseline supervised models. In this paper, we first delve into this phenomenon for an interpretation. We find that MAE tends to focus on low-level domain information during reconstructing pixels while changing the reconstruction target to token features could mitigate this problem. However, not all features are beneficial, as we then find reconstructing high-level features can hardly improve the model's transferability, indicating a trade-off between filtering domain information and preserving the image's global structure. In all, the reconstruction target matters for the CDFSL task. Based on the above findings and interpretations, we further propose Domain-Agnostic Masked Image Modeling (DAMIM) for the CDFSL task. DAMIM includes an Aggregated Feature Reconstruction module to automatically aggregate features for reconstruction, with balanced learning of domain-agnostic information and images' global structure, and a Lightweight Decoder module to further benefit the encoder's generalizability. Experiments on four CDFSL datasets demonstrate that our method achieves state-of-the-art performance.