CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection

TL;DR

This paper introduces CRoFT, a fine-tuning framework for vision-language models that enhances out-of-distribution generalization and open-set detection by optimizing gradient behaviors, supported by theoretical insights and extensive experiments.

Contribution

We propose a novel objective for fine-tuning VL-PTMs that improves OOD generalization and open-set detection through concurrent optimization based on gradient magnitude minimization.

Findings

Our method outperforms existing approaches in OOD detection accuracy.

Theoretical analysis links gradient magnitude minimization to domain-consistent Hessians.

Extensive experiments validate the effectiveness of CRoFT across benchmarks.

Abstract

Recent vision-language pre-trained models (VL-PTMs) have shown remarkable success in open-vocabulary tasks. However, downstream use cases often involve further fine-tuning of VL-PTMs, which may distort their general knowledge and impair their ability to handle distribution shifts. In real-world scenarios, machine learning systems inevitably encounter both covariate shifts (e.g., changes in image styles) and semantic shifts (e.g., test-time unseen classes). This highlights the importance of enhancing out-of-distribution (OOD) generalization on covariate shifts and simultaneously detecting semantic-shifted unseen classes. Thus a critical but underexplored question arises: How to improve VL-PTMs' generalization ability to closed-set OOD data, while effectively detecting open-set unseen classes during fine-tuning? In this paper, we propose a novel objective function of OOD detection that also serves to improve OOD generalization. We show that minimizing the gradient magnitude of energy scores on training data leads to domain-consistent Hessians of classification loss, a strong indicator for OOD generalization revealed by theoretical analysis. Based on this finding, we have developed a unified fine-tuning framework that allows for concurrent optimization of both tasks. Extensive experiments have demonstrated the superiority of our method. The code is available at https://github.com/LinLLLL/CRoFT.