FusionForce: End-to-end Differentiable Neural-Symbolic Layer for Trajectory Prediction

TL;DR

FusionForce introduces an end-to-end differentiable neural-symbolic model that predicts robot trajectories on rough terrain by integrating learnable forces with a physics-enforcing layer, improving generalization and simulation speed.

Contribution

The paper presents a novel neural-symbolic architecture combining learnable forces with a differentiable physics engine for trajectory prediction.

Findings

Reduces sim-to-real gap in robot trajectory prediction.

Achieves $10^4$ trajectories per second in simulation.

Improves out-of-distribution generalization.

Abstract

We propose end-to-end differentiable model that predicts robot trajectories on rough offroad terrain from camera images and/or lidar point clouds. The model integrates a learnable component that predicts robot-terrain interaction forces with a neural-symbolic layer that enforces the laws of classical mechanics and consequently improves generalization on out-of-distribution data. The neural-symbolic layer includes a differentiable physics engine that computes the robot's trajectory by querying these forces at the points of contact with the terrain. As the proposed architecture comprises substantial geometrical and physics priors, the resulting model can also be seen as a learnable physics engine conditioned on real sensor data that delivers $10^4$ trajectories per second. We argue and empirically demonstrate that this architecture reduces the sim-to-real gap and mitigates out-of-distribution sensitivity. The differentiability, in conjunction with the rapid simulation speed, makes the model well-suited for various applications including model predictive control, trajectory shooting, supervised and reinforcement learning, or SLAM.