Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization

TL;DR

This paper introduces a novel model-based deep reinforcement learning framework for dynamic portfolio optimization, integrating prediction, data augmentation, and behavior cloning modules to enhance trading decisions and robustness.

Contribution

It presents a new RL architecture combining prediction, data augmentation, and imitation learning for autonomous trading, improving over prior model-free approaches.

Findings

The proposed model achieves higher profitability than baseline strategies.

It demonstrates robustness and risk sensitivity in real-market simulations.

The architecture works with both on-policy and off-policy RL algorithms.

Abstract

Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architecture with an autonomous trading agent such that, investment decisions and actions are made periodically, based on a global objective, with autonomy. In particular, without relying on a purely model-free RL agent, we train our trading agent using a novel RL architecture consisting of an infused prediction module (IPM), a generative adversarial data augmentation module (DAM) and a behavior cloning module (BCM). Our model-based approach works with both on-policy or off-policy RL algorithms. We further design the back-testing and execution engine which interact with the RL agent in real time. Using historical {\em real} financial market data, we simulate trading with practical constraints, and demonstrate that our proposed model is robust, profitable and risk-sensitive, as compared to baseline trading strategies and model-free RL agents from prior work.