Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets

TL;DR

This paper introduces four risk-sensitive reinforcement learning algorithms tailored for trading markets, addressing challenges like high variance, exploration costs, and multi-agent interactions, with theoretical and practical guarantees.

Contribution

It proposes risk-averse algorithms with variance reduction, extends multi-agent learning to include adversarial perturbations, and pioneers empirical game theory analysis for risk-sensitive payoffs.

Findings

Algorithms perform well under adversarial perturbations

Risk-sensitive methods improve robustness and stability

First to extend empirical game theory to risk-sensitive multi-agent settings

Abstract

Trading markets represent a real-world financial application to deploy reinforcement learning agents, however, they carry hard fundamental challenges such as high variance and costly exploration. Moreover, markets are inherently a multiagent domain composed of many actors taking actions and changing the environment. To tackle these type of scenarios agents need to exhibit certain characteristics such as risk-awareness, robustness to perturbations and low learning variance. We take those as building blocks and propose a family of four algorithms. First, we contribute with two algorithms that use risk-averse objective functions and variance reduction techniques. Then, we augment the framework to multi-agent learning and assume an adversary which can take over and perturb the learning process. Our third and fourth algorithms perform well under this setting and balance theoretical guarantees with practical use. Additionally, we consider the multi-agent nature of the environment and our work is the first one extending empirical game theory analysis for multi-agent learning by considering risk-sensitive payoffs.