Synthesizing Near-Boundary OOD Samples for Out-of-Distribution Detection

TL;DR

SynOOD leverages foundation models to generate challenging boundary-aligned OOD samples, which are used to fine-tune CLIP models, significantly improving out-of-distribution detection near the in-distribution boundary.

Contribution

The paper introduces SynOOD, a novel method that uses foundation models to synthesize boundary-aware OOD samples for enhanced OOD detection in vision-language models.

Findings

Achieves state-of-the-art OOD detection on ImageNet benchmark.

Effectively generates nuanced boundary-aligned OOD samples.

Significantly improves boundary-level discrimination between InD and OOD.

Abstract

Pre-trained vision-language models have exhibited remarkable abilities in detecting out-of-distribution (OOD) samples. However, some challenging OOD samples, which lie close to in-distribution (InD) data in image feature space, can still lead to misclassification. The emergence of foundation models like diffusion models and multimodal large language models (MLLMs) offers a potential solution to this issue. In this work, we propose SynOOD, a novel approach that harnesses foundation models to generate synthetic, challenging OOD data for fine-tuning CLIP models, thereby enhancing boundary-level discrimination between InD and OOD samples. Our method uses an iterative in-painting process guided by contextual prompts from MLLMs to produce nuanced, boundary-aligned OOD samples. These samples are refined through noise adjustments based on gradients from OOD scores like the energy score, effectively sampling from the InD/OOD boundary. With these carefully synthesized images, we fine-tune the CLIP image encoder and negative label features derived from the text encoder to strengthen connections between near-boundary OOD samples and a set of negative labels. Finally, SynOOD achieves state-of-the-art performance on the large-scale ImageNet benchmark, with minimal increases in parameters and runtime. Our approach significantly surpasses existing methods, and the code is available at https://github.com/Jarvisgivemeasuit/SynOOD.