A Research Agenda for Usability and Generalisation in Reinforcement Learning

TL;DR

This paper advocates for developing user-friendly problem description languages in reinforcement learning to improve usability for non-experts and enhance the generalisation capabilities of RL algorithms across diverse problems.

Contribution

It proposes a research agenda focused on creating formal, accessible problem description languages to make RL more usable and promote better generalisation.

Findings

Highlights the usability challenges in current RL practices.

Identifies the lack of standard problem representations as a barrier to generalisation.

Suggests that formal description languages can bridge usability and generalisation issues.

Abstract

It is common practice in reinforcement learning (RL) research to train and deploy agents in bespoke simulators, typically implemented by engineers directly in general-purpose programming languages or hardware acceleration frameworks such as CUDA or JAX. This means that programming and engineering expertise is not only required to develop RL algorithms, but is also required to use already developed algorithms for novel problems. The latter poses a problem in terms of the usability of RL, in particular for private individuals and small organisations without substantial engineering expertise. We also perceive this as a challenge for effective generalisation in RL, in the sense that is no standard, shared formalism in which different problems are represented. As we typically have no consistent representation through which to provide information about any novel problem to an agent, our agents also cannot instantly or rapidly generalise to novel problems. In this position paper, we advocate for a research agenda centred around the use of user-friendly description languages for describing problems, such that (i) users with little to no engineering expertise can formally describe the problems they would like to be tackled by RL algorithms, and (ii) algorithms can leverage problem descriptions to effectively generalise among all problems describable in the language of choice.