CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning

TL;DR

This paper introduces CoT-Segmenter, a novel framework that leverages Chain-of-Thought reasoning with foundation models like GPT-4 to improve out-of-distribution detection in complex road scene semantic segmentation, addressing challenges in densely packed, distant, and large-object scenarios.

Contribution

It proposes a new CoT-based approach utilizing foundation models to enhance OOD detection in dense, complex road scenes, outperforming existing methods on standard and challenging datasets.

Findings

Outperforms state-of-the-art OOD segmentation methods

Effective in densely packed and small object scenarios

Provides interpretable reasoning for OOD detection

Abstract

Effective Out-of-Distribution (OOD) detection is criti-cal for ensuring the reliability of semantic segmentation models, particularly in complex road environments where safety and accuracy are paramount. Despite recent advancements in large language models (LLMs), notably GPT-4, which significantly enhanced multimodal reasoning through Chain-of-Thought (CoT) prompting, the application of CoT-based visual reasoning for OOD semantic segmentation remains largely unexplored. In this paper, through extensive analyses of the road scene anomalies, we identify three challenging scenarios where current state-of-the-art OOD segmentation methods consistently struggle: (1) densely packed and overlapping objects, (2) distant scenes with small objects, and (3) large foreground-dominant objects. To address the presented challenges, we propose a novel CoT-based framework targeting OOD detection in road anomaly scenes. Our method leverages the extensive knowledge and reasoning capabilities of foundation models, such as GPT-4, to enhance OOD detection through improved image understanding and prompt-based reasoning aligned with observed problematic scene attributes. Extensive experiments show that our framework consistently outperforms state-of-the-art methods on both standard benchmarks and our newly defined challenging subset of the RoadAnomaly dataset, offering a robust and interpretable solution for OOD semantic segmentation in complex driving environments.