From Channel Bias to Feature Redundancy: Uncovering the "Less is More" Principle in Few-Shot Learning

TL;DR

This paper uncovers how channel bias causes feature redundancy in few-shot learning, demonstrating that using fewer, more discriminative features improves accuracy, and introduces AFIA to mitigate this issue.

Contribution

It identifies channel bias as a core obstacle in few-shot learning, linking it to feature redundancy, and proposes a simple soft-masking method to improve representation transfer.

Findings

Using 1-5% of top features boosts accuracy significantly.

Feature redundancy stems from confounding feature dimensions.

AFIA effectively mitigates feature redundancy in few-shot tasks.

Abstract

Deep neural networks often fail to adapt representations to novel tasks under distribution shifts, especially when only a few examples are available. This paper identifies a core obstacle behind this failure: channel bias, where networks develop a rigid emphasis on feature dimensions that were discriminative for the source task, but this emphasis is misaligned and fails to adapt to the distinct needs of a novel task. This bias leads to a striking and detrimental consequence: feature redundancy. We demonstrate that for few-shot tasks, classification accuracy is significantly improved by using as few as 1-5% of the most discriminative feature dimensions, revealing that the vast majority are actively harmful. Our theoretical analysis confirms that this redundancy originates from confounding feature dimensions-those with high intra-class variance but low inter-class separability-which are especially problematic in low-data regimes. This "less is more" phenomenon is a defining characteristic of the few-shot setting, diminishing as more samples become available. To address this, we propose a simple yet effective soft-masking method, Augmented Feature Importance Adjustment (AFIA), which estimates feature importance from augmented data to mitigate the issue. By establishing the cohesive link from channel bias to its consequence of extreme feature redundancy, this work provides a foundational principle for few-shot representation transfer and a practical method for developing more robust few-shot learning algorithms.