Portfolio Optimization with 2D Relative-Attentional Gated Transformer

TL;DR

This paper introduces a novel transformer-based reinforcement learning model with relative positional embeddings and gating mechanisms, effectively handling realistic transaction costs in portfolio optimization.

Contribution

The paper proposes the DPGRGT model, combining 2D relative-attentional gated transformers with reinforcement learning to improve portfolio optimization under realistic constraints.

Findings

Outperforms baseline models on U.S. stock data

Handles transaction costs effectively

Demonstrates stable convergence in training

Abstract

Portfolio optimization is one of the most attentive fields that have been researched with machine learning approaches. Many researchers attempted to solve this problem using deep reinforcement learning due to its efficient inherence that can handle the property of financial markets. However, most of them can hardly be applicable to real-world trading since they ignore or extremely simplify the realistic constraints of transaction costs. These constraints have a significantly negative impact on portfolio profitability. In our research, a conservative level of transaction fees and slippage are considered for the realistic experiment. To enhance the performance under those constraints, we propose a novel Deterministic Policy Gradient with 2D Relative-attentional Gated Transformer (DPGRGT) model. Applying learnable relative positional embeddings for the time and assets axes, the model better understands the peculiar structure of the financial data in the portfolio optimization domain. Also, gating layers and layer reordering are employed for stable convergence of Transformers in reinforcement learning. In our experiment using U.S. stock market data of 20 years, our model outperformed baseline models and demonstrated its effectiveness.