An Ergodic Measure for Active Learning From Equilibrium

TL;DR

This paper introduces KL-Ergodic Exploration from Equilibrium (KL-E3), a novel method enabling robots to explore and gather informative data efficiently by integrating stability and ergodic exploration, applicable to high-dimensional systems.

Contribution

The paper develops a hybrid systems controller that combines equilibrium policies with ergodic exploration, extending to high-dimensional states and enabling real-time online learning.

Findings

Successfully maintains Lyapunov attractiveness while exploring.

Generates informative data for Bayesian optimization, model learning, and reinforcement learning.

Demonstrates effectiveness through simulated robotic systems.

Abstract

This paper develops KL-Ergodic Exploration from Equilibrium ($\text{KL-E}^3$), a method for robotic systems to integrate stability into actively generating informative measurements through ergodic exploration. Ergodic exploration enables robotic systems to indirectly sample from informative spatial distributions globally, avoiding local optima, and without the need to evaluate the derivatives of the distribution against the robot dynamics. Using hybrid systems theory, we derive a controller that allows a robot to exploit equilibrium policies (i.e., policies that solve a task) while allowing the robot to explore and generate informative data using an ergodic measure that can extend to high-dimensional states. We show that our method is able to maintain Lyapunov attractiveness with respect to the equilibrium task while actively generating data for learning tasks such, as Bayesian optimization, model learning, and off-policy reinforcement learning. In each example, we show that our proposed method is capable of generating an informative distribution of data while synthesizing smooth control signals. We illustrate these examples using simulated systems and provide simplification of our method for real-time online learning in robotic systems.