3D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering

TL;DR

3D-MoRe introduces a unified framework that leverages foundational models to generate large-scale 3D-language datasets, significantly improving reasoning and response quality in embodied question answering tasks within complex indoor scenes.

Contribution

The paper presents 3D-MoRe, a novel paradigm that combines multi-modal embedding, cross-modal interaction, and language modeling to generate extensive 3D-language datasets for indoor scene understanding.

Findings

Generated 62,000 QA pairs and 73,000 object descriptions across 1,513 scenes.

Achieved a 2.15% improvement in CIDEr score on ScanQA.

Improved CIDEr@0.5 by 1.84% on ScanRefer.

Abstract

With the growing need for diverse and scalable data in indoor scene tasks, such as question answering and dense captioning, we propose 3D-MoRe, a novel paradigm designed to generate large-scale 3D-language datasets by leveraging the strengths of foundational models. The framework integrates key components, including multi-modal embedding, cross-modal interaction, and a language model decoder, to process natural language instructions and 3D scene data. This approach facilitates enhanced reasoning and response generation in complex 3D environments. Using the ScanNet 3D scene dataset, along with text annotations from ScanQA and ScanRefer, 3D-MoRe generates 62,000 question-answer (QA) pairs and 73,000 object descriptions across 1,513 scenes. We also employ various data augmentation techniques and implement semantic filtering to ensure high-quality data. Experiments on ScanQA demonstrate that 3D-MoRe significantly outperforms state-of-the-art baselines, with the CIDEr score improving by 2.15\%. Similarly, on ScanRefer, our approach achieves a notable increase in CIDEr@0.5 by 1.84\%, highlighting its effectiveness in both tasks. Our code and generated datasets will be publicly released to benefit the community, and both can be accessed on the https://3D-MoRe.github.io.