Two Kinds of Learning Algorithms for Continuous-Time VWAP Targeting Execution

TL;DR

This paper develops and compares reinforcement learning and adaptive dynamic programming algorithms for continuous-time VWAP trading execution, providing theoretical guarantees and empirical validation across different market environments.

Contribution

It introduces a relaxed stochastic optimization framework with an explicit Gaussian optimal policy and extends RL to jump processes, with convergence proofs and practical algorithms.

Findings

Both RL and ADP algorithms converge in tested environments.

ADP performs better with strong price impact environments.

RL algorithms learn directly from interactions without model assumptions.

Abstract

The optimal execution problem has always been a continuously focused research issue, and many reinforcement learning (RL) algorithms have been studied. In this article, we consider the execution problem of targeting the volume weighted average price (VWAP) and propose a relaxed stochastic optimization problem with an entropy regularizer to encourage more exploration. We derive the explicit formula of the optimal policy, which is Gaussian distributed, with its mean value being the solution to the original problem. Extending the framework of continuous RL to processes with jumps, we provide some theoretical proofs for RL algorithms. First, minimizing the martingale loss function leads to the optimal parameter estimates in the mean-square sense, and the second algorithm is to use the martingale orthogonality condition. In addition to the RL algorithm, we also propose another learning algorithm: adaptive dynamic programming (ADP) algorithm, and verify the performance of both in two different environments across different random seeds. Convergence of all algorithms has been verified in different environments, and shows a larger advantage in the environment with stronger price impact. ADP is a good choice when the agent fully understands the environment and can estimate the parameters well. On the other hand, RL algorithms do not require any model assumptions or parameter estimation, and are able to learn directly from interactions with the environment.