Reinforcement Learning with Expert Trajectory For Quantitative Trading

TL;DR

This paper introduces a reinforcement learning approach using expert trajectories for quantitative trading, modeling price prediction as an MDP with multiple alpha factors, and demonstrates improved performance on Chinese futures data.

Contribution

The paper proposes a novel reinforcement learning method incorporating expert experience and multiple alpha factors for better trading decision-making.

Findings

Outperforms traditional technical analysis methods.

Shows robustness against noise in financial data.

Effective on Chinese share price index futures.

Abstract

In recent years, quantitative investment methods combined with artificial intelligence have attracted more and more attention from investors and researchers. Existing related methods based on the supervised learning are not very suitable for learning problems with long-term goals and delayed rewards in real futures trading. In this paper, therefore, we model the price prediction problem as a Markov decision process (MDP), and optimize it by reinforcement learning with expert trajectory. In the proposed method, we employ more than 100 short-term alpha factors instead of price, volume and several technical factors in used existing methods to describe the states of MDP. Furthermore, unlike DQN (deep Q-learning) and BC (behavior cloning) in related methods, we introduce expert experience in training stage, and consider both the expert-environment interaction and the agent-environment interaction to design the temporal difference error so that the agents are more adaptable for inevitable noise in financial data. Experimental results evaluated on share price index futures in China, including IF (CSI 300) and IC (CSI 500), show that the advantages of the proposed method compared with three typical technical analysis and two deep leaning based methods.