OpenView: Empowering MLLMs with Out-of-view VQA

TL;DR

This paper introduces OpenView, a novel framework and dataset for out-of-view visual question answering, enabling multimodal models to reason beyond visible image content, significantly improving their performance.

Contribution

The paper presents a four-stage pipeline, a synthetic panoramic dataset, and a benchmark for out-of-view VQA, advancing the ability of MLLMs to understand beyond the visible image frame.

Findings

MLLMs performance improved from 48.6% to 64.1% on average with OpenView.

OpenView dataset enables effective supervised fine-tuning for out-of-view reasoning.

OpenView benchmark provides a new standard for evaluating out-of-view VQA capabilities.

Abstract

Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline to massively generate multi-choice VQA by leveraging panoramic imagery to enable context-rich and spatial-grounded VQA synthesis with free-view framing. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, upon empowered by OpenView, multiple MLLMs can consistently boost their performance, uplifted from 48.6% to 64.1% on average. Code, benchmark, and data will be available at https://github.com/q1xiangchen/OpenView.