The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio

TL;DR

This paper introduces MultiGen, a framework that combines generative models with physics simulators to enable multimodal (audio-visual) simulation for robots, facilitating zero-shot transfer of policies to real-world tasks like pouring.

Contribution

The paper presents MultiGen, a novel approach integrating large-scale generative models into simulators to synthesize realistic audio, advancing multimodal sim-to-real transfer for robotics.

Findings

Effective zero-shot transfer in robot pouring tasks

Realistic audiovisual simulation without real robot data

Bridging the multimodal sim-to-real gap using generative models

Abstract

Robots must integrate multiple sensory modalities to act effectively in the real world. Yet, learning such multimodal policies at scale remains challenging. Simulation offers a viable solution, but while vision has benefited from high-fidelity simulators, other modalities (e.g. sound) can be notoriously difficult to simulate. As a result, sim-to-real transfer has succeeded primarily in vision-based tasks, with multimodal transfer still largely unrealized. In this work, we tackle these challenges by introducing MultiGen, a framework that integrates large-scale generative models into traditional physics simulators, enabling multisensory simulation. We showcase our framework on the dynamic task of robot pouring, which inherently relies on multimodal feedback. By synthesizing realistic audio conditioned on simulation video, our method enables training on rich audiovisual trajectories -- without any real robot data. We demonstrate effective zero-shot transfer to real-world pouring with novel containers and liquids, highlighting the potential of generative modeling to both simulate hard-to-model modalities and close the multimodal sim-to-real gap.