Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning

TL;DR

This paper proposes a novel normalization layer that flattens long-range minima in the loss landscape's representation space, significantly improving cross-domain few-shot learning performance across multiple datasets.

Contribution

It introduces a simple, lightweight normalization layer that achieves long-range flattening of the loss landscape, enhancing transferability and fine-tuning in CDFSL models.

Findings

Outperforms state-of-the-art methods on 8 datasets

Achieves up to 9% accuracy improvement on individual datasets

Effective for CNNs and ViTs

Abstract

Cross-domain few-shot learning (CDFSL) aims to acquire knowledge from limited training data in the target domain by leveraging prior knowledge transferred from source domains with abundant training samples. CDFSL faces challenges in transferring knowledge across dissimilar domains and fine-tuning models with limited training data. To address these challenges, we initially extend the analysis of loss landscapes from the parameter space to the representation space, which allows us to simultaneously interpret the transferring and fine-tuning difficulties of CDFSL models. We observe that sharp minima in the loss landscapes of the representation space result in representations that are hard to transfer and fine-tune. Moreover, existing flatness-based methods have limited generalization ability due to their short-range flatness. To enhance the transferability and facilitate fine-tuning, we introduce a simple yet effective approach to achieve long-range flattening of the minima in the loss landscape. This approach considers representations that are differently normalized as minima in the loss landscape and flattens the high-loss region in the middle by randomly sampling interpolated representations. We implement this method as a new normalization layer that replaces the original one in both CNNs and ViTs. This layer is simple and lightweight, introducing only a minimal number of additional parameters. Experimental results on 8 datasets demonstrate that our approach outperforms state-of-the-art methods in terms of average accuracy. Moreover, our method achieves performance improvements of up to 9\% compared to the current best approaches on individual datasets. Our code will be released.