Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations

TL;DR

This paper introduces Chat2Map, a method that uses multi-ego conversational audio-visual data and deep reinforcement learning to efficiently map unseen 3D environments with reduced visual data processing.

Contribution

It proposes a novel problem of scene mapping from multi-ego conversations and develops an audio-visual RL approach for selective camera control to improve efficiency.

Findings

Outperforms previous mapping methods in accuracy and efficiency

Achieves a superior cost-accuracy tradeoff in scene mapping

Validates approach on both simulated and real-world data

Abstract

Can conversational videos captured from multiple egocentric viewpoints reveal the map of a scene in a cost-efficient way? We seek to answer this question by proposing a new problem: efficiently building the map of a previously unseen 3D environment by exploiting shared information in the egocentric audio-visual observations of participants in a natural conversation. Our hypothesis is that as multiple people ("egos") move in a scene and talk among themselves, they receive rich audio-visual cues that can help uncover the unseen areas of the scene. Given the high cost of continuously processing egocentric visual streams, we further explore how to actively coordinate the sampling of visual information, so as to minimize redundancy and reduce power use. To that end, we present an audio-visual deep reinforcement learning approach that works with our shared scene mapper to selectively turn on the camera to efficiently chart out the space. We evaluate the approach using a state-of-the-art audio-visual simulator for 3D scenes as well as real-world video. Our model outperforms previous state-of-the-art mapping methods, and achieves an excellent cost-accuracy tradeoff. Project: http://vision.cs.utexas.edu/projects/chat2map.