In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization

TL;DR

This paper investigates how layer-wise sharpness, especially in straggler layers, influences the generalization of CLIP models during Robust Fine-Tuning, challenging traditional beliefs about flat minima and proposing new insights for model robustness.

Contribution

It introduces the concept of layer-wise sharpness in straggler layers as a reliable indicator of CLIP's OOD generalization during interpolation, and explores sparsity to improve robustness.

Findings

Layer-wise sharpness correlates with OOD generalization in CLIP.

Sharpness is not a reliable predictor of generalization for modern architectures.

Sparsity in straggler layers mitigates failure modes in RFT.

Abstract

\textit{Zero-shot} models like CLIP are often fine-tuned on a target dataset to improve its accuracy further, but this can compromise out-of-distribution (OOD) robustness. Robust Fine-Tuning (\texttt{RFT} )~\citep{wortsman2021robust}, which interpolates between the \textit{zero-shot} and \textit{fine-tuned} models, has been proposed to address this issue. However, understanding when \texttt{RFT} actually improves OOD error remains limited. In this work, we empirically investigate the robustness of \texttt{RFT} in CLIP models, with a focus on the \textit{sharpness} of the CLIP model during interpolation. First, we demonstrate that while sharpness may not serve as a reliable indicator for predicting the generalization of modern architectures like CLIP on OOD data, this challenges the conventional belief in the generalization benefits of flat minima in foundation models. However, by examining the role of the \textit{straggler layer} phenomenon, we show that, unlike overall sharpness, the \textit{layer-wise} sharpness of \textit{straggler} layers can reliably capture the generalization performance of interpolated CLIP models on OOD data. Our extensive experiments reveal that \textit{layer-wise} sharpness correlates with generalization in OOD accuracy for \texttt{RFT}. Furthermore, we demonstrate that by inducing sparsity in the \textit{straggler} layers, we can mitigate the \textit{failure mode} phenomenon in \texttt{RFT}. To the best of our knowledge, this is the first work to study the role of sharpness in the \textit{success} of interpolation in the weight space of CLIP foundation models. Our code is available at \url{https://github.com/alirezaabdollahpour/CLIP_Mode_Connectivity}.