Robust Grounding with MLLMs Against Occlusion and Small Objects via Language-Guided Semantic Cues

TL;DR

This paper introduces a method leveraging language-guided semantic cues to improve the robustness of Multimodal Large Language Models in crowded scenes with occlusion and small objects.

Contribution

It proposes a Semantic Cue Extractor and a guidance mechanism that incorporate linguistic semantic priors to enhance grounding performance.

Findings

Improved grounding accuracy in crowded scenes.

Semantic cues help mitigate occlusion and small object challenges.

Method outperforms baseline models in robustness tests.

Abstract

While Multimodal Large Language Models (MLLMs) have enhanced grounding capabilities in general scenes, their robustness in crowded scenes remains underexplored. Crowded scenes entail visual challenges (i.e., occlusion and small objects), which impair object semantics and degrade grounding performance. In contrast, language expressions are immune to such degradation and preserve object semantics. In light of these observations, we propose a novel method that overcomes such constraints by leveraging Language-Guided Semantic Cues (LGSCs). Specifically, our approach introduces a Semantic Cue Extractor (SCE) to derive semantic cues of objects from the visual pipeline of an MLLM. We then guide these cues using corresponding text embeddings to produce LGSCs as linguistic semantic priors. Subsequently, they are reintegrated into the original visual pipeline to refine object semantics. Extensive experiments and analyses demonstrate that incorporating LGSCs into an MLLM effectively improves grounding accuracy in crowded scenes.