A Tensor Low-Rank Approximation for Value Functions in Multi-Task Reinforcement Learning

TL;DR

This paper introduces a low-rank tensor approximation method for multi-task reinforcement learning that captures task similarities implicitly, reducing data needs and improving learning efficiency in diverse environments.

Contribution

It proposes a novel low-rank tensor modeling approach for multi-task Q-functions, enabling implicit task similarity inference without explicit task grouping.

Findings

Effective in benchmark inverted pendulums environment

Successful application to wireless communication devices

Reduces data requirements for multi-task learning

Abstract

In pursuit of reinforcement learning systems that could train in physical environments, we investigate multi-task approaches as a means to alleviate the need for massive data acquisition. In a tabular scenario where the Q-functions are collected across tasks, we model our learning problem as optimizing a higher order tensor structure. Recognizing that close-related tasks may require similar actions, our proposed method imposes a low-rank condition on this aggregated Q-tensor. The rationale behind this approach to multi-task learning is that the low-rank structure enforces the notion of similarity, without the need to explicitly prescribe which tasks are similar, but inferring this information from a reduced amount of data simultaneously with the stochastic optimization of the Q-tensor. The efficiency of our low-rank tensor approach to multi-task learning is demonstrated in two numerical experiments, first in a benchmark environment formed by a collection of inverted pendulums, and then into a practical scenario involving multiple wireless communication devices.