Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model

TL;DR

This paper introduces MM-OOD, a multimodal reasoning approach using large language models to improve out-of-distribution detection in images, addressing limitations of text-only methods and enhancing performance on diverse datasets.

Contribution

The paper proposes a novel multimodal pipeline, MM-OOD, leveraging MLLMs for improved near and far OOD detection through multi-round reasoning and a sketch-generate-elaborate framework.

Findings

Significant performance improvements on Food-101 dataset

Validated scalability on ImageNet-1K

Effective detection in both near and far OOD scenarios

Abstract

Out-of-Distribution (OOD) detection is a critical task that has garnered significant attention. The emergence of CLIP has spurred extensive research into zero-shot OOD detection, often employing a training-free approach. Current methods leverage expert knowledge from large language models (LLMs) to identify potential outliers. However, these approaches tend to over-rely on knowledge in the text space, neglecting the inherent challenges involved in detecting out-of-distribution samples in the image space. In this paper, we propose a novel pipeline, MM-OOD, which leverages the multimodal reasoning capabilities of MLLMs and their ability to conduct multi-round conversations for enhanced outlier detection. Our method is designed to improve performance in both near OOD and far OOD tasks. Specifically, (1) for near OOD tasks, we directly feed ID images and corresponding text prompts into MLLMs to identify potential outliers; and (2) for far OOD tasks, we introduce the sketch-generate-elaborate framework: first, we sketch outlier exposure using text prompts, then generate corresponding visual OOD samples, and finally elaborate by using multimodal prompts. Experiments demonstrate that our method achieves significant improvements on widely used multimodal datasets such as Food-101, while also validating its scalability on ImageNet-1K.