Minimizing Embedding Distortion for Robust Out-of-Distribution Performance

TL;DR

This paper proposes a similarity loss method to fine-tune foundational models, effectively reducing embedding distortion to enhance out-of-distribution performance across diverse tasks.

Contribution

The paper introduces a novel similarity loss for fine-tuning that preserves pre-trained embeddings, improving OOD generalization without sacrificing in-distribution accuracy.

Findings

Significant improvement in OOD performance on satellite image classification.

Enhanced face recognition accuracy under domain shift scenarios.

Maintains strong in-distribution performance while reducing embedding distortion.

Abstract

Foundational models, trained on vast and diverse datasets, have demonstrated remarkable capabilities in generalizing across different domains and distributions for various zero-shot tasks. Our work addresses the challenge of retaining these powerful generalization capabilities when adapting foundational models to specific downstream tasks through fine-tuning. To this end, we introduce a novel approach we call "similarity loss", which can be incorporated into the fine-tuning process of any task. By minimizing the distortion of fine-tuned embeddings from the pre-trained embeddings, our method strikes a balance between task-specific adaptation and preserving broad generalization abilities. We evaluate our approach on two diverse tasks: image classification on satellite imagery and face recognition, focusing on open-class and domain shift scenarios to assess out-of-distribution (OOD) performance. We demonstrate that this approach significantly improves OOD performance while maintaining strong in-distribution (ID) performance.