Time-Inhomogeneous Volatility Aversion for Financial Applications of Reinforcement Learning

TL;DR

This paper introduces a new risk metric for reinforcement learning in finance that allows flexible planning of return splits, addressing limitations of traditional risk measures and enhancing decision-making in sequential financial tasks.

Contribution

It proposes a novel risk metric for RL that enables arbitrary planning of return splits, expanding the applicability of risk-aware RL in finance.

Findings

The new metric penalizes reward uncertainty while allowing flexible return planning.

Theoretical analysis of the properties of the proposed objective.

Numerical experiments demonstrate the metric's effectiveness on toy examples.

Abstract

In finance, sequential decision problems are often faced, for which reinforcement learning (RL) emerges as a promising tool for optimisation without the need of analytical tractability. However, the objective of classical RL is the expected cumulated reward, while financial applications typically require a trade-off between return and risk. In this work, we focus on settings where one cares about the time split of the total return, ruling out most risk-aware generalisations of RL which optimise a risk measure defined on the latter. We notice that a preference for homogeneous splits, which we found satisfactory for hedging, can be unfit for other problems, and therefore propose a new risk metric which still penalises uncertainty of the single rewards, but allows for an arbitrary planning of their target levels. We study the properties of the resulting objective and the generalisation of learning algorithms to optimise it. Finally, we show numerical results on toy examples.