Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization

TL;DR

This paper introduces MetaTrader, a bilevel reinforcement learning framework for portfolio optimization that enhances out-of-domain performance and addresses value overestimation in offline RL settings.

Contribution

MetaTrader is the first to explicitly train RL agents for both in-domain and out-of-domain stock trading performance using a bilevel learning approach.

Findings

MetaTrader outperforms existing RL and traditional models on stock datasets.

The bilevel framework improves generalization to data transformations.

The new TD method reduces value overestimation in offline RL.

Abstract

Reinforcement learning (RL) has shown significant promise for sequential portfolio optimization tasks, such as stock trading, where the objective is to maximize cumulative returns while minimizing risks using historical data. However, traditional RL approaches often produce policies that merely memorize the optimal yet impractical buying and selling behaviors within the fixed dataset. These offline policies are less generalizable as they fail to account for the non-stationary nature of the market. Our approach, MetaTrader, frames portfolio optimization as a new type of partial-offline RL problem and makes two technical contributions. First, MetaTrader employs a bilevel learning framework that explicitly trains the RL agent to improve both in-domain profits on the original dataset and out-of-domain performance across diverse transformations of the raw financial data. Second, our approach incorporates a new temporal difference (TD) method that approximates worst-case TD estimates from a batch of transformed TD targets, addressing the value overestimation issue that is particularly challenging in scenarios with limited offline data. Our empirical results on two public stock datasets show that MetaTrader outperforms existing methods, including both RL-based approaches and traditional stock prediction models.