Robo-GS: A Physics Consistent Spatial-Temporal Model for Robotic Arm with Hybrid Representation

TL;DR

This paper introduces Robo-GS, a hybrid spatial-temporal model that combines mesh geometry, Gaussian kernels, and physics attributes for accurate digital twin creation and simulation of robotic arms, improving real-to-simulation transfer.

Contribution

It presents a novel hybrid representation model with a Gaussian-Mesh-Pixel binding technique for differentiable rendering and physics-based simulation of robotic arms.

Findings

Achieves high-fidelity rendering with Gaussian Splatting.

Enables physically plausible interaction simulation.

Provides a fully differentiable pipeline for optimization.

Abstract

Real2Sim2Real plays a critical role in robotic arm control and reinforcement learning, yet bridging this gap remains a significant challenge due to the complex physical properties of robots and the objects they manipulate. Existing methods lack a comprehensive solution to accurately reconstruct real-world objects with spatial representations and their associated physics attributes. We propose a Real2Sim pipeline with a hybrid representation model that integrates mesh geometry, 3D Gaussian kernels, and physics attributes to enhance the digital asset representation of robotic arms. This hybrid representation is implemented through a Gaussian-Mesh-Pixel binding technique, which establishes an isomorphic mapping between mesh vertices and Gaussian models. This enables a fully differentiable rendering pipeline that can be optimized through numerical solvers, achieves high-fidelity rendering via Gaussian Splatting, and facilitates physically plausible simulation of the robotic arm's interaction with its environment using mesh-based methods. The code,full presentation and datasets will be made publicly available at our website https://robostudioapp.com