Physical Reinforcement Learning

TL;DR

This paper explores the use of contrastive local learning networks (CLLNs), low-power analog systems, for reinforcement learning, demonstrating their potential advantages and unique features compared to digital hardware.

Contribution

It adapts Q-learning for CLLNs, showing their capability to perform RL tasks and discussing their advantages and biological relevance over digital systems.

Findings

Successful implementation of Q-learning on CLLNs for simple RL problems

Highlighting the natural fit of policy and value functions in CLLNs

Discussion of safety and secondary goals relevant to biological systems

Abstract

Digital computers are power-hungry and largely intolerant of damaged components, making them potentially difficult tools for energy-limited autonomous agents in uncertain environments. Recently developed Contrastive Local Learning Networks (CLLNs) - analog networks of self-adjusting nonlinear resistors - are inherently low-power and robust to physical damage, but were constructed to perform supervised learning. In this work we demonstrate success on two simple RL problems using Q-learning adapted for simulated CLLNs. Doing so makes explicit the components (beyond the network being trained) required to enact various tools in the RL toolbox, some of which (policy function and value function) are more natural in this system than others (replay buffer). We discuss assumptions such as the physical safety that digital hardware requires, CLLNs can forgo, and biological systems cannot rely on, and highlight secondary goals that are important in biology and trainable in CLLNs, but make little sense in digital computers.