CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model

TL;DR

CUE-M is a novel multimodal search framework that enhances understanding and retrieval accuracy by integrating image context, intent refinement, external APIs, and dynamic filtering, significantly improving performance on knowledge-based and safety benchmarks.

Contribution

This paper introduces CUE-M, a comprehensive multimodal search system that addresses current limitations through a multi-stage pipeline and adaptive filtering, setting new state-of-the-art results.

Findings

Outperforms existing baselines on knowledge-based VQA benchmarks.

Achieves new state-of-the-art results in multimodal retrieval accuracy.

Effectively filters inappropriate responses using dynamic, policy-driven classifiers.

Abstract

The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has revolutionized information retrieval and expanded the practical applications of AI. However, current systems struggle in accurately interpreting user intent, employing diverse retrieval strategies, and effectively filtering unintended or inappropriate responses, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search framework that addresses these challenges through a multi-stage pipeline comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust filtering pipeline combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific concern defined by organizational policies. Extensive experiments on real-word datasets and public benchmarks on knowledge-based VQA and safety demonstrated that CUE-M outperforms baselines and establishes new state-of-the-art results, advancing the capabilities of multimodal retrieval systems.