Towards Inverse Reinforcement Learning for Limit Order Book Dynamics

TL;DR

This paper explores the application of inverse reinforcement learning to infer reward functions of agents in a simulated limit order book environment, demonstrating the effectiveness of Bayesian neural networks in modeling complex, non-linear behaviors.

Contribution

It introduces a novel IRL approach using Bayesian neural networks for modeling non-linear reward functions in financial multi-agent systems.

Findings

BNN IRL outperforms other methods with more samples

GP and BNN can learn linear reward functions

BNN effectively models complex non-linear rewards

Abstract

Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquiring such reward functions through inference, allowing to generalize the resulting policy to states not observed in the past. This paper investigates whether IRL can infer such rewards from agents within real financial stochastic environments: limit order books (LOB). We introduce a simple one-level LOB, where the interactions of a number of stochastic agents and an expert trading agent are modelled as a Markov decision process. We consider two cases for the expert's reward: either a simple linear function of state features; or a complex, more realistic non-linear function. Given the expert agent's demonstrations, we attempt to discover their strategy by modelling their latent reward function using linear and Gaussian process (GP) regressors from previous literature, and our own approach through Bayesian neural networks (BNN). While the three methods can learn the linear case, only the GP-based and our proposed BNN methods are able to discover the non-linear reward case. Our BNN IRL algorithm outperforms the other two approaches as the number of samples increases. These results illustrate that complex behaviours, induced by non-linear reward functions amid agent-based stochastic scenarios, can be deduced through inference, encouraging the use of inverse reinforcement learning for opponent-modelling in multi-agent systems.

Equations

$$exp\sum _i\sum _t\left[Q_{s_{i,t},a_{i,t}}^{{}^{\prime }r}-log\sum _{a^{\prime }}exp(Q_{s_{i,t},{a^{\prime }}_{i,t}}^{{}^{\prime }r})\right]$$

$$p(f,\theta ,\mathcal{D}|X_f)$$

$${\mathcal{L}}_q={\mathbb{E}}_q[\log p(\mathcal{Z}|\mathbf{w})]-\mathrm{KL}[q(\mathbf{w})\Vert p(\mathbf{w})]$$

$$\underset{\theta }{\max }{\mathcal{L}}_{q_{\theta }}$$

$${\mathbf{s}}_t={\left[\begin{array}{ccc}{v}_b^{(1)}(t)& v_a^{(1)}(t)& i^{(\text{EA})}(t)\end{array}\right]}^T\in \mathcal{S}$$

$${\mathbf{a}}_t={\left[\begin{array}{cc}{v}_b^{(EA)}(t)& v_a^{(EA)}(t)\end{array}\right]}^T\in \mathcal{A}$$

$${\pi }^{(n)}(\cdot |{\mathbf{s}}_t;{\tau }^{(n)})$$

$$o_t^{(n)}$$

$$r(s)=N-v_b^{(1)}-v_a^{(1)}$$

$$r(s)=1-e^{-\alpha \ast (N-v_b^{(1)}-v_a^{(1)}-\beta \ast |i^{(\text{EA})}|)}$$

$$\textbf{EVD}=\mathbb{E}\big{[}\sum_{t=0}^{T}\gamma^{t}r(s_{t})|\pi^{*}\big{]}-\mathbb{E}\big{[}\sum_{t=0}^{T}\gamma^{t}r(s_{t})|\hat{\pi}\big{]}$$

Full text

Machine Learning, ICML

1 Introduction

††footnotetext: Opinions expressed in this paper are those of the authors, and do not necessarily reflect the view of JP Morgan.

Limit order books play a central role in the formation of prices for financial securities in exchanges globally. These systems centralize limit orders of price and volume to buy or sell certain securities from large numbers of dealers and investors, matching bids and offers in a transparent process. The dynamics that emerge from this aggregate process (Preis et al., 2006; Cont et al., 2010) of competitive behaviour fall naturally within the scope of multi-agent learning, and understanding them is of paramount importance to liquidity providers (known as market makers) in order to achieve optimal execution of their operations. We focus on the problem of learning the latent reward function of an expert agent from a set of observations of their external actions in the LOB.

Reinforcement learning (RL) (Sutton & Barto, 2018) is a formal framework to study sequential decision-making, particularly relevant for modelling the behaviour of financial agents in environments like the LOB. In particular, RL allows to model their decision-making process as agents interacting with a dynamic environment through policies that seek to maximize their respective cumulative rewards. Inverse reinforcement learning (Russell, 1998) is therefore a powerful framework to analyze and model the actions of such agents, aiming at discovering their latent reward functions: the most "succinct, robust and transferable definition of a task" (Ng et al., 2000). Once learned, such reward functions can be generalized to unobserved regions of the state space, an important advantage over other learning methods.

Overview.

The paper starts at Section 2 by introducing the foundations of IRL, relevant aspects of the maximum causal entropy model used, and useful properties of Gaussian processes and Bayesian neural networks applied to IRL. A formal description of the limit order book model follows. In Section 3, we formulate the one-level LOB as a finite Markov decision process (MDP) and express its dynamics in terms of a Poisson binomial distribution. Finally, we investigate the performance of two well-known IRL methods (maximum entropy IRL and GP-based IRL), and propose and test an additional IRL approach based on Bayesian neural networks. Each IRL method is tested on two versions of the LOB environment, where the reward function of the expert agent may be either a simple linear function of state features, or a more complex and realistic non-linear reward function.

Related Work.

Agent-based models of financial market microstructure are extensively used (Preis et al., 2006; Navarro & Larralde, 2017; Wang & Wellman, 2017). In most setups, mean-field assumptions (Lasry & Lions, 2007) are made to obtain closed form expressions for the dynamics of the complex, multi-agent environment of the exchanges. We make similar assumptions to obtain a tractable finite MDP model for the one-level limit order book.

Previous attempts have been made to model the evolution of the behaviour of large populations over discrete state spaces, combining MDPs with elements of game theory (Yang et al., 2017), using maximum causal entropy inverse reinforcement learning. Their focus had been around social media data. Recently, Hendricks et al. (2017) used IRL in financial market microstructure for modelling the behaviour of the different classes of agents involved in market exchanges (e.g. high-frequency algorithmic market makers, machine traders, human traders and other investors). We draw inspiration from them, and distinguish two types of agents: automatic agents that induce our environment’s dynamics, and active expert agents that trade in such environment. We focus on inferring the expert agents’ objectives from the point of view of an external observer.

Our simplified MDP model could be seen as a variant of the multi-agent Blotto environment (Borel, 1921; Tukey, 1949; Roberson, 2006; Balduzzi et al., 2019). Blotto is a resource distribution game consisting of two opponent armies having each a limited number of soldiers that need to be distributed across multiple areas or battlefields. Each area is won by the army that has the highest number of soldiers. The winner army is the one that has majority over the highest number of battlefields. This environment is often used to model electoral competition problems where parties have a limited budget and need to reach a maximum number of voters. In our environment, only two areas are used (best bid and ask), but the decisions are conditional to a state, hence the MDP could be seen as a contextual 2-area Blotto variant.

Contributions.

We propose a multi-agent LOB model which provides the possibility of obtaining transition probabilities in closed form, enabling the use of model-based IRL, without giving up reasonable proximity to real world LOB settings. The reward functions we propose have a clear financial interpretation, and allow flexible and comparable testing of different IRL methods in a setup that can be scaled to higher dimension versions of the environment. We then provide results for the three IRL methods discussed: maximum entropy IRL, Gaussian process-based IRL, and our IRL approach based on Bayesian neural networks. Results show that BNNs are able to recover the target rewards, outperforming comparable methods both in IRL performance and in terms of computational efficiency.

2 Background

2.1 Inverse Reinforcement Learning

We base our IRL experiments on a Markov decision process consisting of a tuple $\langle\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,P_{0}\rangle$. $\mathcal{S}$ represents the state space; $\mathcal{A}$ the set of eligible actions; $\mathcal{T}$ represents the model’s transition dynamics, where $\mathcal{T}(s^{\prime},a,s)=p(s^{\prime}|s,a)$ is the transition probability to state $s^{\prime}$ from state $s$ through action $a$; $r(s,a)$ is the unknown reward function we intend to recover; the discount factor $\gamma$ takes values in [0, 1]; and $P_{0}$ represents the initial state distribution.

Reinforcement learning aims under its general forward formulation at finding an optimal policy $\pi^{*}$ that maximizes the expected cumulative reward \mathbb{E}\big{[}\sum_{t=0}^{T}\gamma^{t}r(s_{t})|\pi^{*}\big{]}, where the state-action pairs induced by policy $\pi^{*}$ and transition dynamics $\mathcal{T}$ are denoted in a sequence or trajectory $\mathbf{x}=\{(s_{t},a_{t})\}_{t=0}^{T}$.

On the other hand, inverse reinforcement learning aims at recovering through inference (see Figure 1) an unknown reward function $r(s,a)$, where we assume $\pi^{*}(a|s)$ to be an optimal policy from where a collection of expert demonstrations $\mathcal{D}=\{\mathbf{x}_{n}\}_{n=1}^{N}$ is drawn. However, one of the main difficulties of the IRL problem is its ill-posed nature, as there could be more than one optimal policy that explains a given set of demonstrations $\mathcal{D}$ (Ng et al., 1999). This ambiguity is handled by the maximum entropy framework (Ziebart et al., 2008).

Maximum Causal Entropy Model.

The principle of maximum causal entropy (Ziebart, 2010) assumes that the distribution of trajectories generated by the optimal policy follows $\pi^{*}(a|s)\propto\exp\left\{Q^{\prime}\left(s_{t},a_{t}\right)\right\}$ (Ziebart et al., 2008; Haarnoja et al., 2017), where $Q^{\prime}$ is a is a soft Q-function that incorporates the treatment of sequentially-revealed information, regularizing the objective of forward reinforcement learning with respect to differential entropy $\mathcal{H}(\cdot)$ as described in Fu et al. (2017): $Q^{\prime}\left(s_{t},a_{t}\right)=r_{t}(s,a)+\mathbb{E}_{\left(s_{t+1},\ldots\right)\sim\pi}\left[\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}}\left(r\left(s_{t^{\prime}},a_{t^{\prime}}\right)+\mathcal{H}\left(\pi\left(\cdot|s_{t^{\prime}}\right)\right)\right)\right]$.

The inference problem for an unknown reward function $r(s,a)$ thus boils down to maximizing the likelihood $P(\mathcal{D}|r)$ as in Levine et al. (2011):

[TABLE]

2.2 IRL methods considered

The three inverse reinforcement learning methods that we will test on our LOB model for both linear and exponential expert rewards are: maximum entropy IRL (MaxEnt), Gaussian processes-based IRL (GPIRL), and our implementation through Bayesian neural networks (BNN IRL). These methods are defined as follows:

Maximum Entropy IRL.

Proposed by Ziebart (2010), this method builds on the maximum causal entropy objective function described above, combined with a linearity assumption on the structure of the reward as a function of the state features: $\textbf{r}(s)=\textbf{w}^{T}\phi(s)$, where $\phi(s):\mathcal{S}\mapsto\mathbb{R}^{m}$ is the $m$-dimensional mapping from the state to the feature vector. The IRL problem then boils down to the inference of such w.

Gaussian Process-based IRL.

Maximum causal entropy provides a method to infer values of the reward function on specific points of the state space. This is enough for cases where the MDP is finite and where the observed demonstrations cover all the state space, which are not very common. In this context, it is important to learn a structure of rewards that mirrors as closely as possible the behaviour of the expert agent.

While many prior IRL methods assume linearity of the reward function, GP-based IRL (Levine et al., 2011), expands the function space of possible inferred rewards to non-linear reward structures. Under this model, the reward function $r$ is modeled by a zero-mean GP prior with a kernel or covariance function $k_{\theta}$. If $X_{f}\in\mathbb{R}^{n,m}$ is the feature matrix defining a finite number $n$ of $m$-dimensional states, and $f$ is the reward function evaluated at $X_{f}$: $f=r(X_{f})$, then $f|X,\theta\sim\mathcal{N}(0,K_{f,f})$ where $[K_{f,f}]_{i,j}=k_{\theta}(x_{i},x_{j})$.

The GPIRL objective (Levine et al., 2011) is then maximized with respect to the finite rewards $f$ and $\theta$:

[TABLE]

Inside the above integral, we can recognize the IRL objective $p(\mathcal{D}|r)$, the GP posterior $p(r|f,\theta,X_{f})$ and the prior of $f$ and $\theta$. To mitigate the intractability of this integral, Levine et al. (2011) use the Deterministic Training Conditional (DTC) approximation, which reduces the GP posterior to its mean function.

Bayesian Neural Networks applied to IRL.

A neural network (NN) is a superposition of multiple layers consisting of linear transformations and non-linear functions. Infinitely wide NNs whose weights are random variables have been proven to converge to a GP (Neal, 1995; Williams & Jacobs, 1997) and hence represent a universal function approximator (Cybenko, 1989). However, this convergence property is not applicable to finite NNs, which are the networks used in practice. Bayesian neural networks are equivalent networks in the finite case. BNNs have been the focus of multiple studies (Neal, 1995; MacKay, 1992; Gal & Ghahramani, 2016) and are known for their useful regularization properties. Since exact inference is computationally challenging, variational inference has instead been used to approximate these models (Hinton & Van Camp, 1993; Peterson, 1987; Graves, 2011; Gal & Ghahramani, 2016).

Given a dataset $\mathcal{Z}=\{(x_{n},y_{n})\}_{n=1}^{N}$, we wish to learn the mapping $\{(x_{n}\rightarrow y_{n})\}_{n=1}^{N}$ in a robust way. A BNN can be characterized through a prior distribution $p(\mathbf{w})$ placed on its weights, and the likelihood $p(\mathcal{Z}|\mathbf{w})$. An approximate posterior distribution $q(\mathbf{w})$ can be fit through Bayesian variational inference, maximizing the evidence lower bound (ELBO):

[TABLE]

We parametrize $q(\mathbf{w})$ with $\theta$ and use the mean-field variational inference approximation (Peterson, 1987), which assumes that the weights of each layer factorize to independent distributions. We choose a diagonal Gaussian prior distribution $p(\mathbf{w})$ and solve the optimization problem:

[TABLE]

In the context of the IRL problem, we leverage the benefits of BNNs to generalize point estimates provided by maximum causal entropy to a reward function in a robust and efficient way. Unlike in GPIRL, where the full objective is maximized at each iteration, BNN IRL consists of two separate steps:

• •

Inference step: we first optimize the IRL objective $p(\mathcal{D}|\hat{r})$, obtaining a finite number of point estimates of the reward $\{\hat{r}(s_{n})=\hat{r}_{n}\}_{n=1}^{N}$ over a finite number of states $\{s_{n}\in\mathcal{S}\}_{n=1}^{N}$

• •

Learning step: secondly, we use these point estimates to train a BNN that learns the mapping between state features $s\in\mathcal{S}$ and their respective inferred reward values $\hat{r}(s)\in\mathbb{R}$.

The number of point estimates used is the number of states existing in the expert’s demonstrations. This means that we do not use each state in the state space once, but as many times as it exists in the demonstrations $\mathcal{D}$. This is equivalent to tailoring the learning rate of the Bayesian neural network to match the state visitation counts.

2.3 Limit Order Book

Our experimental setup builds on limit order books (LOBs): here we introduce some basic definitions following the conventions of Gould et al. (2013) and Zhang et al. (2019). Two types of orders exist in an LOB: bids (buy orders) and asks (sell orders). A bid order for an asset is a quote to buy the underlying asset at or below a specified price $\mathbf{P}_{b}(t)$. Conversely, the ask order is a quote to sell the asset at or above a certain price $\mathbf{P}_{a}(t)$. For a certain price, the bid and ask orders have respective sizes (volumes) $\mathbf{V}_{b}(t)$ and $\mathbf{V}_{a}(t)$. Each level ($l$) of the LOB is represented by one pair ($p^{(l)}_{b}(t),v^{(l)}_{b}(t)$) or ($p^{(l)}_{a}(t),v^{(l)}_{a}(t)$), generally ranked in decreasing order of competitiveness.

3 Experiments

Experimental Setup.

Our environment setup is a one-level LOB, resulting from the interaction of $N$ (Markovian) trading agents (TA) and an expert agent (EA trader). Without loss of generality, the EA is regulated to have up to $I_{\text{max}}$ inventory (amount of securities in their portfolio). The EA is unaware of the strategies of each of the other trading agents, but adapts to them through trial and error, as in reinforcement learning frameworks (Sutton & Barto, 2018). The EA solves the following MDP $\langle\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,P_{0}\rangle$:

• •

State Space $\mathcal{S}$. The environment state at time-step $t$, $\mathbf{s}_{t}$, is a 3-dimensional vector:

[TABLE]

where $v^{(1)}_{b}(t)$, $v^{(1)}_{a}(t)\in\mathbb{R}_{+}$ are the volumes available in the one-level LOB through the best bid and ask levels, respectively, and $i^{(\text{EA})}(t)\in\{-I_{\text{max}},\ldots,0,\ldots,+I_{\text{max}}\}$ is the inventory held by the expert agent at time-step $t$.

• •

Action Space $\mathcal{A}$. The expert agent chooses to take an action $\mathbf{a}_{t}$ at time-step $t$, by selecting volumes $v^{(EA)}_{b}(t)$ and $v^{(EA)}_{a}(t)$ to match the trading objectives defined by their reward with those available in the LOB through the other traders, at the best bid and ask, respectively:

[TABLE]

where $v^{(EA)}_{b}(t)+v^{(EA)}_{a}(t)\leq N$ (see Transition Dynamics below). The EA is here an active market participant, which actively sells at the best ask and buys at the best bid, while the trading agents on the other side of the LOB only place passive orders.

• •

Transition Dynamics $\mathcal{T}$. At each time-step $t$, the $n$-th trader, $n\in\{1,..,N\}$, places a single order, $o^{(n)}_{t}$ at either side of the LOB (best bid, or best ask). The traders follow stochastic policies:

[TABLE]

where $o^{(n)}_{t}=1$ corresponds to a trader placing an order at the best bid, and $o^{(n)}_{t}=0$ at the best ask. By construction, each of the trading agents has to place exactly one order, a bid or an ask, at each time-step. Ber($\cdot$) denotes the Bernoulli distribution. The temperature parameters $\tau=(\tau_{1},\ldots,\tau_{n})$ are independent and unknown to the expert agent.

Hence, the aggregate intermediate bid orders that were generated by the N trading agents, $v^{(TA)}_{b}(t)$ are the sum of $N$ independent Bernoulli variables, whose parameter is conditioned on the environment state $\mathbf{s}_{t}$ and the idiosyncratic temperature parameters $\tau$.

Therefore, the number of intermediate bids is distributed according to a Poisson binomial distribution (Shepp & Olkin, 1981), whose probability mass function can be expressed in closed form. Since each of the trading agents (but not the EA) should place exactly one order, the aggregate intermediate ask orders are $v^{(TA)}_{a}(t)=N-v^{(TA)}_{b}(t)$. The aggregate bids and asks of the trading agents form an intermediate state called the belief state $\mathbf{b}_{t}=(v^{(TA)}_{b}(t),v^{(TA)}_{a}(t))$.

Although the EA can only see the last available LOB snapshot contained in $\mathbf{s}_{t}$, their orders ($\mathbf{a}_{t}$) are executed against the intermediate state $\mathbf{b}_{t}$. This could be compared in real world LOBs to a slippage effect. Based on the number of bids and asks finally matched by the EA in the LOB, their inventory is updated as a net long or short position. Considering the above dynamics of the LOB, the transition matrix $\mathcal{T}$, can be calculated exactly. This allows for an exact solution of the MDP. An illustration of the dynamics is provided in Figure 2.

• •

Reward Function $r$. The selection of the reward function is crucial, since it induces the behaviour of the agent (Berger, 2013). The reward function can be either a function of state $\mathbf{s}_{t}$ and action $\mathbf{a}_{t}$; or equivalently, following the dynamics $\mathcal{T}$, of the next state $\mathbf{s}_{t+1}$, (Sutton & Barto, 2018). We follow the latter convention, and use the following two reward functions for the expert agent to test each IRL method considered:

Linear reward, equal to the EA’s hit count:

[TABLE]

This reward is equivalent to the hit count of the EA because there are always exactly N orders in the intermediate state $\mathbf{b}_{t}$, and therefore the next state $\mathbf{s}_{t}$ resulting from the execution of the EA’s active orders $\mathbf{a}_{t}$ against the TA orders $\mathbf{b}_{t}$, can be effectively seen as the remaining orders, i.e. the number of orders out of a maximum of N that the EA was not able to match.

This reward function does not incentivize directly to sell inventory. However, since the episode is terminated when maximum inventory $I_{\text{max}}$ is exceeded, the market maker is implicitly motivated not to violate this constraint, since the simulation will then be terminated and the cumulative reward will be reduced. 2. 2.

Exponential reward is a model widely used in economic theory to characterize levels of risk aversion (Pratt, 1964). We may use it to define a non-linear reward function that depends on both the inventory of the EA and their hit count:

[TABLE]

where $\alpha,\beta\in\mathbb{R}_{\geq 0}$ are chosen based on the level of risk aversion of the agent.

Both reward structures are illustrated in Figures 3- 4.

Results and Discussion.

Our experiments are performed on synthetic data, where $N=3$ is the number of trading agents with temperatures $\tau=(0.1,0.5,1.0)$, and $I_{\text{max}}=5$. We also consider an episodic setup, where the simulation is terminated either when the maximum inventory is violated, or $T=5$ time-steps are completed. The initial state is uniformly sampled from the non-terminal states in $\mathcal{S}$.

• •

Performance metric. Following previous IRL literature (Jin et al., 2017; Wulfmeier et al., 2015) we evaluate the performance of each method through their respective Expected Value Differences (EVD). EVD quantifies the difference between:

the expected cumulative reward earned by following the optimal policy $\pi^{*}$ implied by the true rewards; 2. 2.

the expected cumulative reward earned by following the policy $\hat{\pi}$ implied by the rewards inferred through IRL.

[TABLE]

Since the expert’s observed behaviour could have been generated by different reward functions, we compare the EVD yielded by inferred rewards per method, rather than directly comparing each inferred reward against the ground truth reward.

We run two versions of our experiments, where the expert agent has either a linear or an exponential reward function. Each IRL method is run for 512, 1024, 2048, 4096, 8192 and 16384 demonstrations. The results obtained are presented in Figure 5: as expected, all three IRL methods tested (MaxEnt IRL, GPIRL, BNN-IRL), learn fairly well linear reward functions. However, for an agent with an exponential reward, GPIRL and BNN-IRL are able to discover the latent function significantly better, with BNN outperforming as the number of demonstrations increases. In addition to improved EVD, our BNN-IRL experiments provide a significant improvement in computational time as compared to GPIRL, hence enabling potentially more efficient scalability of IRL on LOBs to state spaces of higher dimensions.

4 Conclusions

In this paper we attempt an application of IRL to a stochastic, non-stationary financial environment (the LOB) whose competing agents may follow complex reward functions. While as expected all the methods considered are able to recover linear latent reward functions, only GP-based IRL (Levine et al., 2011) and our implementation through BNNs are able to recover more realistic non-linear expert rewards, thus mitigating most of the challenges imposed by this stochastic multi-agent environment. Moreover, our BNN method outperforms GPIRL for larger numbers of demonstrations, and is less computationally intensive. This may enable future work to study LOBs of higher dimensions, and on increasingly realistic number and complexity of agents involved.