GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields

TL;DR

GaussFly introduces a contrastive reinforcement learning framework that decouples scene representation from policy learning, enabling efficient and robust visuomotor control for aerial vehicles in complex 3D environments.

Contribution

It proposes a novel real-to-sim-to-real paradigm using 3D Gaussian Splatting and contrastive learning to improve sim-to-real transfer for aerial vehicle policies.

Findings

Achieves superior sample efficiency in simulated and real environments.

Enables zero-shot transfer to unseen real-world environments.

Outperforms baseline methods in asymptotic performance.

Abstract

Learning visuomotor policies for Autonomous Aerial Vehicles (AAVs) relying solely on monocular vision is an attractive yet highly challenging paradigm. Existing end-to-end learning approaches directly map high-dimensional RGB observations to action commands, which frequently suffer from low sample efficiency and severe sim-to-real gaps due to the visual discrepancy between simulation and physical domains. To address these long-standing challenges, we propose GaussFly, a novel framework that explicitly decouples representation learning from policy optimization through a cohesive real-to-sim-to-real paradigm. First, to achieve a high-fidelity real-to-sim transition, we reconstruct training scenes using 3D Gaussian Splatting (3DGS) augmented with explicit geometric constraints. Second, to ensure robust sim-to-real transfer, we leverage these photorealistic simulated environments and employ contrastive representation learning to extract compact, noise-resilient latent features from the rendered RGB images. By utilizing this pre-trained encoder to provide low-dimensional feature inputs, the computational burden on the visuomotor policy is significantly reduced while its resistance against visual noise is inherently enhanced. Extensive experiments in simulated and real-world environments demonstrate that GaussFly achieves superior sample efficiency and asymptotic performance compared to baselines. Crucially, it enables robust and zero-shot policy transfer to unseen real-world environments with complex textures, effectively bridging the sim-to-real gap.