Explore and Tell: Embodied Visual Captioning in 3D Environments

TL;DR

This paper introduces Embodied Captioning, a task where an agent navigates 3D environments to generate detailed scene descriptions, supported by a new dataset and a novel model that outperforms baselines.

Contribution

It proposes the Embodied Captioning task, creates the ET-Cap dataset, and develops the CaBOT model combining navigation and captioning for comprehensive scene understanding.

Findings

CaBOT outperforms baseline models in descriptive accuracy

The ET-Cap dataset contains 10K annotated 3D scenes

Navigation improves caption quality in complex environments

Abstract

While current visual captioning models have achieved impressive performance, they often assume that the image is well-captured and provides a complete view of the scene. In real-world scenarios, however, a single image may not offer a good viewpoint, hindering fine-grained scene understanding. To overcome this limitation, we propose a novel task called Embodied Captioning, which equips visual captioning models with navigation capabilities, enabling them to actively explore the scene and reduce visual ambiguity from suboptimal viewpoints. Specifically, starting at a random viewpoint, an agent must navigate the environment to gather information from different viewpoints and generate a comprehensive paragraph describing all objects in the scene. To support this task, we build the ET-Cap dataset with Kubric simulator, consisting of 10K 3D scenes with cluttered objects and three annotated paragraphs per scene. We propose a Cascade Embodied Captioning model (CaBOT), which comprises of a navigator and a captioner, to tackle this task. The navigator predicts which actions to take in the environment, while the captioner generates a paragraph description based on the whole navigation trajectory. Extensive experiments demonstrate that our model outperforms other carefully designed baselines. Our dataset, codes and models are available at https://aim3-ruc.github.io/ExploreAndTell.