Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets

TL;DR

This paper introduces elliptic uncertainty sets for robust reinforcement learning in finance, effectively modeling market impact and improving trading performance under real-world conditions.

Contribution

It develops a novel class of elliptic uncertainty sets with closed-form solutions, enabling efficient robust policy evaluation in financial markets.

Findings

Achieves higher Sharpe ratios in trading tasks.

Remains robust with increasing trade volumes.

Provides scalable and realistic market impact modeling.

Abstract

In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices. However, during deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact. This mismatch between training and deployment environments can significantly degrade performance. Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties, but typically rely on symmetric structures that fail to capture the directional nature of market impact. To address this issue, we develop a novel class of elliptic uncertainty sets. We establish both implicit and explicit closed-form solutions for the worst-case uncertainty under these sets, enabling efficient and tractable robust policy evaluation. Experiments on single-asset and multi-asset trading tasks demonstrate that our method achieves superior Sharpe ratio and remains robust under increasing trade volumes, offering a more faithful and scalable approach to RL in financial markets.