LMM-Det: Make Large Multimodal Models Excel in Object Detection

TL;DR

LMM-Det demonstrates that large multimodal models can be effectively adapted for object detection tasks without specialized detection modules, by analyzing and optimizing their capabilities through data and inference adjustments.

Contribution

Proposes a simple approach to enable large multimodal models to perform object detection without additional detection-specific components.

Findings

Significant recall degradation in LMMs for object detection compared to specialist detectors.

Data distribution adjustment and inference optimization improve detection recall.

Extensive experiments validate the effectiveness of LMM-Det.

Abstract

Large multimodal models (LMMs) have garnered wide-spread attention and interest within the artificial intelligence research and industrial communities, owing to their remarkable capability in multimodal understanding, reasoning, and in-context learning, among others. While LMMs have demonstrated promising results in tackling multimodal tasks like image captioning, visual question answering, and visual grounding, the object detection capabilities of LMMs exhibit a significant gap compared to specialist detectors. To bridge the gap, we depart from the conventional methods of integrating heavy detectors with LMMs and propose LMM-Det, a simple yet effective approach that leverages a Large Multimodal Model for vanilla object Detection without relying on specialized detection modules. Specifically, we conduct a comprehensive exploratory analysis when a large multimodal model meets with object detection, revealing that the recall rate degrades significantly compared with specialist detection models. To mitigate this, we propose to increase the recall rate by introducing data distribution adjustment and inference optimization tailored for object detection. We re-organize the instruction conversations to enhance the object detection capabilities of large multimodal models. We claim that a large multimodal model possesses detection capability without any extra detection modules. Extensive experiments support our claim and show the effectiveness of the versatile LMM-Det. The datasets, models, and codes are available at https://github.com/360CVGroup/LMM-Det.