Towards Multimodal Multitask Scene Understanding Models for Indoor Mobile Agents

TL;DR

This paper introduces MMISM, a multimodal multitask model for indoor scene understanding in mobile agents, capable of handling multiple outputs and modalities, addressing data scarcity and fusion challenges, and outperforming single-task models.

Contribution

The paper presents MMISM, a novel multi-modality, multi-task indoor scene understanding model that effectively fuses RGB and Lidar data for diverse perception tasks.

Findings

MMISM achieves comparable or better performance than single-task models.

Improves 3D object detection results by 11.7% on ARKitScenes dataset.

Addresses key challenges in indoor scene understanding for mobile agents.

Abstract

The perception system in personalized mobile agents requires developing indoor scene understanding models, which can understand 3D geometries, capture objectiveness, analyze human behaviors, etc. Nonetheless, this direction has not been well-explored in comparison with models for outdoor environments (e.g., the autonomous driving system that includes pedestrian prediction, car detection, traffic sign recognition, etc.). In this paper, we first discuss the main challenge: insufficient, or even no, labeled data for real-world indoor environments, and other challenges such as fusion between heterogeneous sources of information (e.g., RGB images and Lidar point clouds), modeling relationships between a diverse set of outputs (e.g., 3D object locations, depth estimation, and human poses), and computational efficiency. Then, we describe MMISM (Multi-modality input Multi-task output Indoor Scene understanding Model) to tackle the above challenges. MMISM considers RGB images as well as sparse Lidar points as inputs and 3D object detection, depth completion, human pose estimation, and semantic segmentation as output tasks. We show that MMISM performs on par or even better than single-task models; e.g., we improve the baseline 3D object detection results by 11.7% on the benchmark ARKitScenes dataset.