Composable Model-Free RL for Navigation with Input-Affine Systems

TL;DR

This paper introduces a composable, model-free reinforcement learning approach for robot navigation that learns and combines value functions for different environment elements, providing formal safety guarantees and improved performance.

Contribution

It develops a novel continuous-time HJB-based framework and a model-free actor-critic algorithm for composing obstacle avoidance and goal-reaching policies.

Findings

Achieves formal obstacle-avoidance guarantees via QCQP composition.

Demonstrates improved navigation performance over PPO baseline.

Provides a model-free alternative to control barrier functions.

Abstract

As autonomous robots move into complex, dynamic real-world environments, they must learn to navigate safely in real time, yet anticipating all possible behaviors is infeasible. We propose a composable, model-free reinforcement learning method that learns a value function and an optimal policy for each individual environment element (e.g., goal or obstacle) and composes them online to achieve goal reaching and collision avoidance. Assuming unknown nonlinear dynamics that evolve in continuous time and are input-affine, we derive a continuous-time Hamilton-Jacobi-Bellman (HJB) equation for the value function and show that the corresponding advantage function is quadratic in the action and optimal policy. Based on this structure, we introduce a model-free actor-critic algorithm that learns policies and value functions for static or moving obstacles using gradient descent. We then compose multiple reach/avoid models via a quadratically constrained quadratic program (QCQP), yielding formal obstacle-avoidance guarantees in terms of value-function level sets, providing a model-free alternative to CLF/CBF-based controllers. Simulations demonstrate improved performance over a PPO baseline applied to a discrete-time approximation.