PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks

TL;DR

This paper introduces PID-inspired history encoding architectures for deep reinforcement learning in partially observable control tasks, improving robustness and performance across tracking and locomotion benchmarks.

Contribution

It proposes novel PID-inspired history encoders for deep RL, enhancing robustness and performance in partially observable control tasks.

Findings

Encoders produce more robust policies than prior methods.

Achieve 1.7x better average performance on locomotion tasks.

Improve tracking task performance with PID-based features.

Abstract

Deep reinforcement learning (RL) has shown immense potential for learning to control systems through data alone. However, one challenge deep RL faces is that the full state of the system is often not observable. When this is the case, the policy needs to leverage the history of observations to infer the current state. At the same time, differences between the training and testing environments makes it critical for the policy not to overfit to the sequence of observations it sees at training time. As such, there is an important balancing act between having the history encoder be flexible enough to extract relevant information, yet be robust to changes in the environment. To strike this balance, we look to the PID controller for inspiration. We assert the PID controller's success shows that only summing and differencing are needed to accumulate information over time for many control tasks. Following this principle, we propose two architectures for encoding history: one that directly uses PID features and another that extends these core ideas and can be used in arbitrary control tasks. When compared with prior approaches, our encoders produce policies that are often more robust and achieve better performance on a variety of tracking tasks. Going beyond tracking tasks, our policies achieve 1.7x better performance on average over previous state-of-the-art methods on a suite of locomotion control tasks.