MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

TL;DR

This paper introduces MultiOOD, a new multimodal OOD detection benchmark, and proposes A2D and NP-Mix methods that significantly improve OOD detection performance across diverse datasets and modalities.

Contribution

It establishes the first multimodal OOD benchmark and introduces novel training algorithms, A2D and NP-Mix, to enhance OOD detection in multimodal settings.

Findings

Adding modalities improves OOD detection accuracy.

A2D encourages modality prediction discrepancy, correlating with better OOD detection.

NP-Mix synthesizes outliers using nearest neighbor features, boosting performance.

Abstract

Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. Our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.