Unified Sensor Simulation for Autonomous Driving

TL;DR

XSIM is a comprehensive sensor simulation framework for autonomous driving that improves geometric and appearance modeling, addressing specific challenges of spherical sensors like LiDARs, and achieves state-of-the-art results on multiple datasets.

Contribution

The paper introduces XSIM, a unified sensor simulation framework with novel phase modeling and Gaussian representation techniques tailored for autonomous driving sensors.

Findings

Outperforms recent baselines on Waymo, Argoverse 2, PandaSet datasets.

Provides more accurate and photorealistic sensor simulations.

Enhances geometric consistency in simulated sensor data.

Abstract

In this work, we introduce \textbf{XSIM}, a sensor simulation framework for autonomous driving. XSIM extends 3DGUT splatting with a generalized rolling-shutter modeling tailored for autonomous driving applications. Our framework provides a unified and flexible formulation for appearance and geometric sensor modeling, enabling rendering of complex sensor distortions in dynamic environments. We identify spherical cameras, such as LiDARs, as a critical edge case for existing 3DGUT splatting due to cyclic projection and time discontinuities at azimuth boundaries leading to incorrect particle projection. To address this issue, we propose a phase modeling mechanism that explicitly accounts temporal and shape discontinuities of Gaussians projected by the Unscented Transform at azimuth borders. In addition, we introduce an extended 3D Gaussian representation that incorporates two distinct opacity parameters to resolve mismatches between geometry and color distributions. As a result, our framework provides enhanced scene representations with improved geometric consistency and photorealistic appearance. We evaluate our framework extensively on multiple autonomous driving datasets, including Waymo Open Dataset, Argoverse 2, and PandaSet. Our framework consistently outperforms strong recent baselines and achieves state-of-the-art performance across all datasets. The source code is publicly available at \href{https://github.com/whesense/XSIM}{https://github.com/whesense/XSIM}.