Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency

TL;DR

This paper introduces a geometric knowledge-guided framework that calibrates biased feature distributions in latent space, improving federated learning and long-tailed recognition by leveraging transferability of features from foundation models.

Contribution

It proposes a novel distribution calibration method using geometric shapes of features from foundation models, applicable to federated learning and long-tailed recognition.

Findings

Effective in bridging local and global data gaps in federated learning.

Improves tail class recognition in long-tailed datasets.

Boosts performance across multiple benchmarks.

Abstract

Despite the fast progress of deep learning, one standing challenge is the gap of the observed training samples and the underlying true distribution. There are multiple reasons for the causing of this gap e.g. sampling bias, noise etc. In the era of foundation models, we show that when leveraging the off-the-shelf (vision) foundation models (e.g., CLIP, DINOv2) for feature extraction, the geometric shapes of the resulting feature distributions exhibit remarkable transferability across domains and datasets. To verify its practical usefulness, we embody our geometric knowledge-guided distribution calibration framework in two popular and challenging settings: federated learning and long-tailed recognition. In the federated setting, we devise a technique of acquiring the global geometric shape under privacy constraints, then leverage this knowledge to generate new samples for clients, in the aim of bridging the gap between local and global observations. In long-tailed learning, it utilizes the geometric knowledge transferred from sample-rich categories to recover the true distribution for sample-scarce tail classes. Comprehensive experiments show that our proposed geometric knowledge-guided distribution calibration effectively overcomes information deficits caused by data heterogeneity and sample imbalance, with boosted performance across benchmarks.