FineFT: Efficient and Risk-Aware Ensemble Reinforcement Learning for Futures Trading

TL;DR

FineFT introduces a three-stage ensemble reinforcement learning framework that enhances profitability and reduces risk in high-leverage crypto futures trading by improving convergence and incorporating capability-aware risk management.

Contribution

The paper presents a novel ensemble RL method with selective updates and VAE-based capability boundary detection tailored for high-leverage futures trading, addressing stability and risk concerns.

Findings

Outperforms 12 SOTA baselines in 6 financial metrics.

Reduces risk by more than 40% in high-frequency crypto futures trading.

Improves convergence and stability through selective ensemble updates and VAE-guided routing.

Abstract

Futures are contracts obligating the exchange of an asset at a predetermined date and price, notable for their high leverage and liquidity and, therefore, thrive in the Crypto market. RL has been widely applied in various quantitative tasks. However, most methods focus on the spot and could not be directly applied to the futures market with high leverage because of 2 challenges. First, high leverage amplifies reward fluctuations, making training stochastic and difficult to converge. Second, prior works lacked self-awareness of capability boundaries, exposing them to the risk of significant loss when encountering new market state (e.g.,a black swan event like COVID-19). To tackle these challenges, we propose the Efficient and Risk-Aware Ensemble Reinforcement Learning for Futures Trading (FineFT), a novel three-stage ensemble RL framework with stable training and proper risk management. In stage I, ensemble Q learners are selectively updated by ensemble TD errors to improve convergence. In stage II, we filter the Q-learners based on their profitabilities and train VAEs on market states to identify the capability boundaries of the learners. In stage III, we choose from the filtered ensemble and a conservative policy, guided by trained VAEs, to maintain profitability and mitigate risk with new market states. Through extensive experiments on crypto futures in a high-frequency trading environment with high fidelity and 5x leverage, we demonstrate that FineFT outperforms 12 SOTA baselines in 6 financial metrics, reducing risk by more than 40% while achieving superior profitability compared to the runner-up. Visualization of the selective update mechanism shows that different agents specialize in distinct market dynamics, and ablation studies certify routing with VAEs reduces maximum drawdown effectively, and selective update improves convergence and performance.