UniCoRN: Unified Commented Retrieval Network with LMMs

TL;DR

UniCoRN is a unified multimodal retrieval and generation framework that enhances complex visual question answering by integrating retrieval and language models, demonstrating significant improvements on retrieval and commenting tasks.

Contribution

It introduces an entity adapter module and a new Commented Retrieval task with dataset, combining retrieval and generation in a single frozen LMM.

Findings

+4.5% recall over state-of-the-art in composed retrieval

+14.9% METEOR / +18.4% BEM over RAG in commenting

Effective integration of retrieval and generation capabilities

Abstract

Multimodal retrieval methods have limitations in handling complex, compositional queries that require reasoning about the visual content of both the query and the retrieved entities. On the other hand, Large Multimodal Models (LMMs) can answer with language to more complex visual questions, but without the inherent ability to retrieve relevant entities to support their answers. We aim to address these limitations with UniCoRN, a Unified Commented Retrieval Network that combines the strengths of composed multimodal retrieval methods and generative language approaches, going beyond Retrieval-Augmented Generation (RAG). We introduce an entity adapter module to inject the retrieved multimodal entities back into the LMM, so it can attend to them while generating answers and comments. By keeping the base LMM frozen, UniCoRN preserves its original capabilities while being able to perform both retrieval and text generation tasks under a single integrated framework. To assess these new abilities, we introduce the Commented Retrieval task (CoR) and a corresponding dataset, with the goal of retrieving an image that accurately answers a given question and generate an additional textual response that provides further clarification and details about the visual information. We demonstrate the effectiveness of UniCoRN on several datasets showing improvements of +4.5% recall over the state of the art for composed multimodal retrieval and of +14.9% METEOR / +18.4% BEM over RAG for commenting in CoR.