Reinforcement Learning for Agents with Many Sensors and Actuators Acting in Categorizable Environments

TL;DR

This paper introduces a categorizability assumption for reinforcement learning in complex robotic environments, enabling faster learning by focusing on relevant sensors and actions, validated through simulated robotic tasks.

Contribution

It formalizes the categorizability property and presents a novel algorithm that leverages this assumption to improve learning efficiency in multi-sensor, multi-actuator agents.

Findings

Reduced learning time compared to existing algorithms

Effective in landmark navigation and gait generation tasks

Validated on simulated realistic robotic problems

Abstract

In this paper, we confront the problem of applying reinforcement learning to agents that perceive the environment through many sensors and that can perform parallel actions using many actuators as is the case in complex autonomous robots. We argue that reinforcement learning can only be successfully applied to this case if strong assumptions are made on the characteristics of the environment in which the learning is performed, so that the relevant sensor readings and motor commands can be readily identified. The introduction of such assumptions leads to strongly-biased learning systems that can eventually lose the generality of traditional reinforcement-learning algorithms. In this line, we observe that, in realistic situations, the reward received by the robot depends only on a reduced subset of all the executed actions and that only a reduced subset of the sensor inputs (possibly different in each situation and for each action) are relevant to predict the reward. We formalize this property in the so called 'categorizability assumption' and we present an algorithm that takes advantage of the categorizability of the environment, allowing a decrease in the learning time with respect to existing reinforcement-learning algorithms. Results of the application of the algorithm to a couple of simulated realistic-robotic problems (landmark-based navigation and the six-legged robot gait generation) are reported to validate our approach and to compare it to existing flat and generalization-based reinforcement-learning approaches.