Enhancing Multi-Image Question Answering via Submodular Subset Selection

TL;DR

This paper introduces a method to improve multi-image question answering by selecting relevant images using submodular functions, enhancing retrieval efficiency and scalability in large image collections.

Contribution

It proposes a novel submodular subset selection approach to enhance retriever frameworks for multi-image question answering tasks.

Findings

Improved retrieval performance with larger image sets.

Effective use of anchor-based queries for better subset selection.

Enhanced scalability in multi-image reasoning tasks.

Abstract

Large multimodal models (LMMs) have achieved high performance in vision-language tasks involving single image but they struggle when presented with a collection of multiple images (Multiple Image Question Answering scenario). These tasks, which involve reasoning over large number of images, present issues in scalability (with increasing number of images) and retrieval performance. In this work, we propose an enhancement for retriever framework introduced in MIRAGE model using submodular subset selection techniques. Our method leverages query-aware submodular functions, such as GraphCut, to pre-select a subset of semantically relevant images before main retrieval component. We demonstrate that using anchor-based queries and augmenting the data improves submodular-retriever pipeline effectiveness, particularly in large haystack sizes.