A Design Trajectory Map of Human-AI Collaborative Reinforcement Learning Systems: Survey and Taxonomy

TL;DR

This paper provides a comprehensive survey and introduces a novel design trajectory map for human-AI collaborative reinforcement learning systems, aiming to guide future research and system design.

Contribution

It offers the first systematic taxonomy and a new modeling tool, the Human-AI CRL Design Trajectory Map, for designing and analyzing collaborative RL frameworks.

Findings

Developed a systematic taxonomy of CRL frameworks.

Created the Human-AI CRL Design Trajectory Map as a modeling tool.

Identified key challenges and future research directions in CRL.

Abstract

Driven by the algorithmic advancements in reinforcement learning and the increasing number of implementations of human-AI collaboration, Collaborative Reinforcement Learning (CRL) has been receiving growing attention. Despite this recent upsurge, this area is still rarely systematically studied. In this paper, we provide an extensive survey, investigating CRL methods based on both interactive reinforcement learning algorithms and human-AI collaborative frameworks that were proposed in the past decade. We elucidate and discuss via synergistic analysis methods both the growth of the field and the state-of-the-art; we conceptualise the existing frameworks from the perspectives of design patterns, collaborative levels, parties and capabilities, and review interactive methods and algorithmic models. Specifically, we create a new Human-AI CRL Design Trajectory Map, as a systematic modelling tool for the selection of existing CRL frameworks, as well as a method of designing new CRL systems, and finally of improving future CRL designs. Furthermore, we elaborate generic Human-AI CRL challenges, providing the research community with a guide towards novel research directions. The aim of this paper is to empower researchers with a systematic framework for the design of efficient and 'natural' human-AI collaborative methods, making it possible to work on maximised realisation of humans' and AI's potentials.