Learning Policies for Continuous Control via Transition Models

TL;DR

This paper introduces a neural network-based approach that learns transition models from interaction data to improve continuous control policies, aiming for more human-like motor control in robots.

Contribution

It presents a modular neural network architecture that learns system dynamics and control policies simultaneously using deep active inference principles.

Findings

Outperforms linear quadratic regulator baseline in control tasks

Demonstrates effective learning of system dynamics from prediction errors

Provides a pathway toward human-like motor control in robots

Abstract

It is doubtful that animals have perfect inverse models of their limbs (e.g., what muscle contraction must be applied to every joint to reach a particular location in space). However, in robot control, moving an arm's end-effector to a target position or along a target trajectory requires accurate forward and inverse models. Here we show that by learning the transition (forward) model from interaction, we can use it to drive the learning of an amortized policy. Hence, we revisit policy optimization in relation to the deep active inference framework and describe a modular neural network architecture that simultaneously learns the system dynamics from prediction errors and the stochastic policy that generates suitable continuous control commands to reach a desired reference position. We evaluated the model by comparing it against the baseline of a linear quadratic regulator, and conclude with additional steps to take toward human-like motor control.