MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception

TL;DR

MITO introduces a comprehensive mmWave dataset and simulator that enable perception of objects through occlusions, facilitating advancements in non-line-of-sight perception using diverse data and synthetic image generation.

Contribution

This work provides the first large-scale mmWave dataset with high-resolution synthetic aperture images and an open-source simulator for NLOS perception research.

Findings

Over 24 million mmWave frames collected

550 high-resolution synthetic aperture images generated

Benchmarks for NLOS segmentation and classification established

Abstract

The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment. However, optical perception can often be disrupted due to common occurrences, such as occlusions, which can pose challenges to existing vision systems. We present MITO, the first millimeter-wave (mmWave) dataset of diverse, everyday objects, collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Unlike visible light, mmWave signals can penetrate common occlusions (e.g., cardboard boxes, fabric, plastic) but each mmWave frame has much lower resolution than typical cameras. To capture higher-resolution mmWave images, we leverage the robot's mobility and fuse frames over the synthesized aperture. MITO captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave (synthetic aperture) images in line-of-sight and non-light-of-sight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals, taken from 76 different objects. We develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh. Finally, we demonstrate the utility of our dataset and simulator for enabling broader NLOS perception by developing benchmarks for NLOS segmentation and classification.