Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning

TL;DR

This paper investigates how disrupting image token continuity in Vision Transformers affects cross-domain few-shot learning, revealing that less continuity encourages reliance on smaller patterns, thus improving generalization across distant domains.

Contribution

It uncovers the role of token continuity in ViT's domain generalization and proposes a simple method to disrupt continuity, enhancing performance in cross-domain few-shot tasks.

Findings

Disrupting token continuity reduces domain gaps.

Smaller patterns are more transferable across domains.

The proposed method outperforms state-of-the-art approaches.

Abstract

Vision Transformer (ViT) has achieved remarkable success due to its large-scale pretraining on general domains, but it still faces challenges when applying it to downstream distant domains that have only scarce training data, which gives rise to the Cross-Domain Few-Shot Learning (CDFSL) task. Inspired by Self-Attention's insensitivity to token orders, we find an interesting phenomenon neglected in current works: disrupting the continuity of image tokens (i.e., making pixels not smoothly transited across patches) in ViT leads to a noticeable performance decline in the general (source) domain but only a marginal decrease in downstream target domains. This questions the role of image tokens' continuity in ViT's generalization under large domain gaps. In this paper, we delve into this phenomenon for an interpretation. We find continuity aids ViT in learning larger spatial patterns, which are harder to transfer than smaller ones, enlarging domain distances. Meanwhile, it implies that only smaller patterns within each patch could be transferred under extreme domain gaps. Based on this interpretation, we further propose a simple yet effective method for CDFSL that better disrupts the continuity of image tokens, encouraging the model to rely less on large patterns and more on smaller ones. Extensive experiments show the effectiveness of our method in reducing domain gaps and outperforming state-of-the-art works. Codes and models are available at https://github.com/shuaiyi308/ReCIT.