Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation

TL;DR

Asset Harvester is a comprehensive pipeline that converts sparse real-world driving observations into complete, simulation-ready 3D assets for autonomous vehicle simulation and testing.

Contribution

It introduces a system-level design combining data curation, geometry-aware preprocessing, and a novel 3D generation model to produce complete 3D assets from limited-view AV data.

Findings

Enables scalable conversion of sparse AV observations into 3D assets.

Addresses real-world data challenges with SparseViewDiT model.

Improves simulation asset quality for autonomous vehicle development.

Abstract

Closed-loop simulation is a core component of autonomous vehicle (AV) development, enabling scalable testing, training, and safety validation before real-world deployment. Neural scene reconstruction converts driving logs into interactive 3D environments for simulation, but it does not produce complete 3D object assets required for agent manipulation and large-viewpoint novel-view synthesis. To address this challenge, we present Asset Harvester, an image-to-3D model and end-to-end pipeline that converts sparse, in-the-wild object observations from real driving logs into complete, simulation-ready assets. Rather than relying on a single model component, we developed a system-level design for real-world AV data that combines large-scale curation of object-centric training tuples, geometry-aware preprocessing across heterogeneous sensors, and a robust training recipe that couples sparse-view-conditioned multiview generation with 3D Gaussian lifting. Within this system, SparseViewDiT is explicitly designed to address limited-angle views and other real-world data challenges. Together with hybrid data curation, augmentation, and self-distillation, this system enables scalable conversion of sparse AV object observations into reusable 3D assets.