Derivation of the Boltzmann Equation for Financial Brownian Motion: Direct Observation of the Collective Motion of High-Frequency Traders

TL;DR

This paper develops a microscopic model of high-frequency traders' collective behavior in financial markets, deriving Boltzmann-like equations and validating them with empirical data, bridging microscopic dynamics with market-level phenomena.

Contribution

It introduces the first microscopic model of HFTs validated by direct data analysis, linking microscopic trader behavior to macroscopic market dynamics.

Findings

Quantitative empirical law of traders' trend-following behavior

Derivation of Boltzmann-like and Langevin-like equations for HFTs

Model validation showing agreement with market data

Abstract

A microscopic model is established for financial Brownian motion from the direct observation of the dynamics of high-frequency traders (HFTs) in a foreign exchange market. Furthermore, a theoretical framework parallel to molecular kinetic theory is developed for the systematic description of the financial market from microscopic dynamics of HFTs. We report first on a microscopic empirical law of traders' trend-following behavior by tracking the trajectories of all individuals, which quantifies the collective motion of HFTs but has not been captured in conventional order-book models. We next introduce the corresponding microscopic model of HFTs and present its theoretical solution paralleling molecular kinetic theory: Boltzmann-like and Langevin-like equations are derived from the microscopic dynamics via the Bogoliubov-Born-Green-Kirkwood-Yvon hierarchy. Our model is the first microscopic model that has been directly validated through data analysis of the microscopic dynamics, exhibiting quantitative agreements with mesoscopic and macroscopic empirical results.

Equations

$$⟨\Delta z_i{⟩}_{\Delta p}\approx c_i\tanh \frac{\Delta p}{\Delta p_i^{\ast }},V_{\Delta p}[\Delta z_i]\approx {\sigma }_i^2,$$

$$\frac{d{z}_i(t)}{dt}=c\tanh \frac{\Delta p(t)}{\Delta p^{\ast }}+\sigma {\eta }_i^{\mathrm{R}}(t)$$

$$p(t+0)=b_i(t),\Delta p(t+0)=b_i(t)-p(t),$$

$$z_i(t+0)=z_i(t)-\frac{L_i}{2},z_j(t+0)=z_j(t)+\frac{L_j}{2}.$$

$${\varphi }_{L{L}^{\prime }}(r,r^{\prime })\approx {\varphi }_L(r){\varphi }_{L^{\prime }}(r^{\prime }),$$

$$\frac{\partial {\varphi }_L}{\partial t}\approx \frac{{\sigma }^2}{2}\frac{{\partial }^2{\varphi }_L}{\partial r^2}+N\sum _{s=\pm 1}\int d{L}^{\prime }\rho (L^{\prime })[{\tilde{J}}_{L{L}^{\prime }}^s(r+sL/2)-{\tilde{J}}_{L{L}^{\prime }}^s]$$

$$\Delta p(T+1)=c\tau (T)\tanh \frac{\Delta p(T)}{\Delta p^{\ast }}+\zeta (T),$$

$$\rho (L)=\frac{L^3}{6L^{\ast 4}}{e}^{-L/L^{\ast }}$$

$$f_{\mathrm{A}}(r)=\frac{4e^{-\frac{3r}{2L^{\ast }}}}{3L^{\ast }}\left[\left(2+\frac{r}{L^{\ast }}\right)\sinh \frac{r}{2L^{\ast }}-\frac{r{e}^{-\frac{r}{2L^{\ast }}}}{2L^{\ast }}\right].$$

$$P(\ge |\Delta p|;\kappa )\approx e^{-|\Delta p|/\kappa }(|\Delta p|\to \infty )$$

$$P^{\mathrm{w}}(\ge |\Delta p|)={\int }_0^{\infty }d\kappa Q(\kappa )P^{2h}(\ge |\Delta p|;\kappa )\propto |\Delta p{|}^{-m}$$

$$\begin{array}{rl}\frac{d{z}_i}{dt}& =c\tanh \frac{\Delta p}{\Delta p^{\ast }}+\sigma {\eta }_i^{\mathrm{R}}+{\eta }_i^{\mathrm{T}},{\eta }_i^{\mathrm{T}}\equiv \sum _{k=1}^{\infty }\sum _j^{j\ne i}\Delta z_{ij}\delta (t-{\tau }_{k;ij}),\\ \frac{dp}{dt}& =\sum _{k=1}^{\infty }\sum _{i,j}^{i<j}(p^{\mathrm{post}}-p)\delta (t-{\tau }_{k;ij}),\frac{d\Delta p}{dt}=\sum _{k=1}^{\infty }\sum _{i,j}^{i<j}(\Delta p^{\mathrm{post}}-\Delta p)\delta (t-{\tau }_{k;ij}),\end{array}$$

$$|z_i({\tau }_{k;ij})-z_j({\tau }_{k;ij})|=\frac{L_i+L_j}{2}⟹\Delta z_{ij}=-\frac{L_i}{2}\mathrm{sgn}(z_i-z_j),p^{\mathrm{post}}=z_i+\Delta z_{ij},\Delta p^{\mathrm{post}}\equiv z_i+\Delta z_{ij}-p$$

$$z_i(t+dt)-z_i(t)=\{\begin{array}{ll}0& (\text{mbox}Probability=1-\lambda dt)\\ c\Delta t_{\mathrm{can}}\tanh \frac{\Delta p}{\Delta p^{\ast }}+\sigma \sqrt{\Delta t_{\mathrm{can}}}{\eta }_i^{\mathrm{R}}& (\text{mbox}Probability=\lambda dt)\end{array}$$

$$z_{c.m.}\equiv \frac{1}{N}\sum _{i=1}^N{z}_i,r_i\equiv z_i-z_{c.m.}.$$

$$\frac{d{z}_{c.m.}}{dt}=c\tanh \frac{\Delta p}{\Delta p^{\ast }}+\xi ,\frac{d{r}_i}{dt}=\sigma {\eta }_i^{\mathrm{R}}+{\eta }_i^{\mathrm{T}}-\xi ,\xi \equiv \frac{1}{N}\sum _{j=1}^N\left(\sigma {\eta }_j^{\mathrm{R}}+{\eta }_j^{\mathrm{T}}\right).$$

$$\frac{d{r}_i}{dt}\approx \sigma {\eta }_i^{\mathrm{R}}+{\eta }_i^{\mathrm{T}}.$$

$$\frac{d{r}_1}{dt}=\sigma {\eta }_{1;\epsilon }^{\mathrm{R}}+\sum _{k=1}^{\infty }\Delta r_1\delta (t-{\tau }_k),\frac{d{r}_2}{dt}=\sigma {\eta }_{2;\epsilon }^{\mathrm{R}}+\sum _{k=1}^{\infty }\Delta r_2\delta (t-{\tau }_k)$$

$$|r_1({\tau }_k)-r_2({\tau }_k)|=\frac{L_1+L_2}{2}⟹\Delta r_1=-\frac{L_1}{2}\mathrm{sgn}(r_1-r_2),\Delta r_2=-\frac{L_2}{2}\mathrm{sgn}(r_2-r_1).$$

$$\frac{\partial P_{12}}{\partial t}=\sum _{i=1,2}\frac{{\sigma }^2}{2}\frac{{\partial }^2{P}_{12}}{\partial r_i^2}+\sum _{s=\pm 1}\frac{{\sigma }^2}{2}\left[\delta (r_1-r_2)|{\tilde{\partial }}_{12}|P_{12}\left(r_1+\frac{s{L}_1}{2},r_2-\frac{s{L}_2}{2}\right)-\delta \left(r_1-r_2-s\frac{L_1+L_2}{2}\right)|{\tilde{\partial }}_{12}|P_{12}\right],$$

$$\frac{df(r_1,r_2)}{dt}$$

$$=\sum _{i=1,2}\sigma {\eta }_{i;\epsilon }^{\mathrm{R}}\frac{\partial f(r_1,r_2)}{\partial r_i}+\sigma \left[f(r_1+\Delta r_1,r_2+\Delta r_2)-f(r_1,r_2)\right]\delta \left(|r_1-r_2|-\frac{L_1+L_2}{2}\right)|{\eta }_{1;\epsilon }^{\mathrm{R}}-{\eta }_{2;\epsilon }^{\mathrm{R}}|,$$

$$\frac{df}{dt}=\sum _{i=1,2}\sigma {\eta }_{i;\epsilon }^{\mathrm{R}}\frac{\partial f}{\partial r_i}+\sum _{s=\pm 1}s\sigma \left[f\left(r_1-\frac{s{L}_1}{2},r_2+\frac{s{L}_2}{2}\right)-f\right]\delta \left(r_1-r_2-s\frac{L_1+L_2}{2}\right)({\eta }_{1;\epsilon }^{\mathrm{R}}-{\eta }_{2;\epsilon }^{\mathrm{R}}).$$

$$\bigg{<}\frac{df}{dt}\bigg{>}=\sum_{i=1,2}\sigma\bigg{<}\eta^{\mathrm{R}}_{i;\varepsilon}\frac{\partial f}{\partial r_{i}}\bigg{>}+\sum_{s=\pm 1}s\sigma\bigg{<}\left[f\left(r_{1}-\frac{sL_{1}}{2},r_{2}+\frac{sL_{2}}{2}\right)-f\right]\delta\left(r_{1}-r_{2}-s\frac{L_{1}+L_{2}}{2}\right)(\eta^{\mathrm{R}}_{1;\varepsilon}-\eta^{\mathrm{R}}_{2;\varepsilon})\bigg{>}.$$

$$\frac{\partial P_{12}}{\partial t}=\sum _{i=1,2}\frac{{\sigma }^2}{2}\frac{{\partial }^2{P}_{12}}{\partial x_i^2}+\sum _{s=\pm 1}\frac{s{\sigma }^2}{2}\left[-\delta (x_1-x_2){\tilde{\partial }}_{12}{P}_{12}\left(x_1+\frac{s{L}_1}{2},x_2-\frac{s{L}_2}{2}\right)+\delta \left(x_1-x_2-s\frac{L_1+L_2}{2}\right){\tilde{\partial }}_{12}{P}_{12}\right]$$

$$\underset{\epsilon \to 0}{\lim }⟨{\eta }_{i;\epsilon }^{\mathrm{R}}(t)g(r_1(t),r_2(t))⟩=\underset{\epsilon \to 0}{\lim }{\int }_0^{t}ds⟨{\eta }_{i;\epsilon }^{\mathrm{R}}(t){\eta }_{i;\epsilon }^{\mathrm{R}}(s)⟩⟨\frac{\delta g(r_1(t),r_2(t))}{\delta {\eta }_{i;\epsilon }^{\mathrm{R}}(s)}⟩=\frac{\sigma }{2}⟨\frac{\partial g(r_1,r_2)}{\partial r_i}⟩.$$

$$s\tilde{\partial}_{12}P_{12}(x_{1},x_{2})\bigg{|}_{x_{1}-x_{2}=s(L_{1}+L_{2})/2}=-|\tilde{\partial}_{12}|P_{12}(x_{1},x_{2})\bigg{|}_{x_{1}-x_{2}=s(L_{1}+L_{2})/2}.$$

$$\displaystyle\frac{\partial P_{1}(r_{1})}{\partial t}=\frac{\sigma^{2}}{2}\frac{\partial^{2}P_{1}(r_{1})}{\partial r_{1}^{2}}+\sum_{s=\pm 1}\left[J_{12}^{s}(r_{1}+sL_{1}/2)-J_{12}^{s}(r_{1})\right],\>\>\>J_{12}^{s}(r)\equiv\frac{\sigma^{2}}{2}|\tilde{\partial}_{12}|P_{12}(r_{1},r_{2})\bigg{|}_{r_{1}-r_{2}=s(L_{1}+L_{2})/2},$$

$$\frac{\partial {\varphi }_L(r)}{\partial t}=\frac{{\sigma }^2}{2}\frac{{\partial }^2{\varphi }_L(r)}{\partial r^2}+C({\varphi }_{L{L}^{\prime }})$$

$$C(\phi_{LL^{\prime}})=N\sum_{s=\pm 1}\int dL^{\prime}\rho(L^{\prime})\left[J^{s}_{LL^{\prime}}(r+sL/2)-J^{s}_{LL^{\prime}}(r)\right],\>\>\>J^{s}_{LL^{\prime}}(r)=\frac{\sigma^{*2}}{2}|\tilde{\partial}_{rr^{\prime}}|\phi_{LL^{\prime}}(r,r^{\prime})\bigg{|}_{r-r^{\prime}=s(L+L^{\prime})/2}$$

$${\varphi }_{L{L}^{\prime }}(r,r^{\prime })\approx {\varphi }_L(r){\varphi }_{L^{\prime }}(r^{\prime }).$$

Full text

Introduction.– In physics, the study of colloidal Brownian motion has a long history beginning with Einstein’s famous work Einstein1905 ; the understanding of its mechanism has been systematically developed in kinetic theory Chapman1970 ; VanKampen . Specifically, from microscopic Newtonian dynamics, the Boltzmann and Langevin equations are derived for the mesoscopic and macroscopic dynamics, respectively. This framework is a rigid foundation for various nonequilibrium systems (e.g., active matter, granular gas, Feynman ratchets, and traffic flow Bertin2006 ; Brilliantov ; Broech2004 ; Broeck2006 ; Helbing ; Nishinari ; Prigogine ), and its direct experimental foundation has been revisited because of recent technological breakthroughs Raizen2011 ; Raizen2010 .

In light of this success, it is natural to apply this framework beyond physics to social science Pareschi2013 , such as finance. Indeed, the concept of random walks was historically invented for price dynamics by Bachelier earlier than Einstein Bachelier , and its similarities to physical Brownian motion (e.g., the fluctuation-dissipation relation) are intensively studied by recent high-frequency data analysis Yura2014 . As an idea in statistical physics, the dynamics of financial markets are expected to be clarified from first principles by extending kinetic theory.

Although this idea is attractive, the kinetic description has not been established for financial Brownian motion. Why has not this idea been realized yet? In our view, the biggest problem is the absence of established microscopic models; there exist empirical validations of mesoscopic Slanina2014 ; Maslov2000 ; Daniels2003 ; Smith2003 ; Bouchaud2002 ; Farmer2005 ; Yura2014 and macroscopic models Mantegna1995 ; Mantegna2000 ; Plerou1999 ; Lux1996 ; Guillaume1997 ; Longin1996 ; PUCK , whereas no microscopic model has been validated by direct empirical analysis. Indeed, previous microscopic models Kyle1985 ; Takayasu1992 ; Bak1997 ; Lux1999 ; Yamada2009 were purely theoretical and have no quantitative evidence microscopically. To overcome this crucial problem as an empirical science, two missing links have to be connected: (i) establishment of the microscopic model by direct observation of traders’ dynamics (Fig. 1a) and (ii) construction of a kinetic theory to show its consistency with mesoscopic and macroscopic findings (i.e., the order-book and price dynamics (Fig. 1b, c)).

In this Letter, we present the corresponding solutions by direct observation of high-frequency trader (HFT) dynamics in a foreign exchange (FX) market: (i) a microscopic model of HFTs is established by direct microscopic evidence, and (ii) corresponding kinetic theory is developed to show its consistency with mesoscopic and macroscopic evidence. We analyzed order-book data with anonymized trader identifiers (IDs) to track trajectories of all individuals. We found an empirical law concerning trend following among HFTs, which has not been captured by previous order-book models. Remarkably, this property induces the collective motion of the order book and naturally leads the layered order-book structure Yura2014 . We then introduce a corresponding microscopic model of trend-following HFTs. Starting from their “equations of motion,” Boltzmann-like and Langevin-like equations are derived for the order-book and price dynamics. A quantitative agreement is finally shown with our empirical all findings. Our work opens the door to systematic descriptions of finance based on microscopic evidence.

Observed microscopic dynamics.– We analyzed the high-frequency FX data between the U.S. dollar (USD) and the Japanese Yen (JPY) on Electronic Broking Services for a week in June 2016 (see Appendix A.1). The currency unit used in this study is 0.001 yen, called the tenth pip (tpip). Here we particularly focused on the dynamics of HFTs Menkveld2011 , frequently submitting or canceling orders according to algorithms (see Appendix A.2). The typical trajectories of bid and ask quoted prices are illustrated in Fig. 2a–c for the top 3 HFTs. They modify their quoted prices by successive submission and cancellation at high speed typically within seconds; almost 99% of their submissions were finally canceled without transactions (see Appendix A.4). With the two-sided quotes they also play the role of liquidity providers Menkveld2013 ; EBSRule according to the market rule, keeping the balance between the bid and ask order book. Buy-sell spreads, the difference between the best bid and ask prices for a single HFT, were observed to fluctuate around certain time constants (see the insets for their distributions and Appendix A.5).

We then report the empirical microscopic law for the trend-following strategy of individual traders. The bid and ask quoted prices of the top $i$th HFT are denoted by $b_{i}$ and $a_{i}$ (see Appendix A.6). We investigated the average movement of the trader’s quoted midprice $z_{i}\equiv(b_{i}+a_{i})/2$ between transactions conditional on the previous market transacted price movement (Fig. 2d). Here we introduce the tick time $T$ as an integer time incremented by every transaction. The mean transaction interval is $9.3$ seconds during this week. Because typical HFTs frequently modify their price between transactions, we here study HFTs’ trend following at one-tick precision. For the top 20 HFTs (Fig. 3), we found that the average and variance of movement $\Delta z_{i}(T)\equiv z_{i}(T+1)-z_{i}(T)$ obeyed

[TABLE]

where the conditional average $\langle\dots\rangle_{\Delta p}$ is taken when the last price change is $\Delta p(T-1)\equiv p(T)-p(T-1)$ and $\Delta z_{i}\neq 0$ (see Appendix A.6) and the conditional variance is defined by $V_{\Delta p}[\Delta z_{i}]\equiv\langle\left(\Delta z_{i}-\langle\Delta z_{i}\rangle_{\Delta p}\right)^{2}\rangle_{\Delta p}$. Here, $p(T)$ is the market transacted price at the $T$ tick, and $c_{i},\Delta p_{i}^{*},\sigma_{i}^{2}$ are characteristic constants unique to the trader and independent of $\Delta p$. Their typical values were found to be $c_{i}\approx 6.0$ tpip, $\Delta p_{i}^{*}\approx 7.5$ tpip, and $\sigma_{i}\approx 14.5$ tpip. Our finding (1) implies that the reaction of traders is linear for small trends but saturates for large trends, and quantifies the collective motion of HFTs. Remarkably, a similar behavior was reported from a price movement data analysis at one-month precision Bouchaud2014 .

Microscopic model.–

Here we introduce a minimal microscopic model of HFTs incorporating the above characters. We make four assumptions: (i) The number of traders is sufficiently large; (ii) traders always quote both bid and ask prices (for the $i$th trader, $b_{i}$ and $a_{i}$) simultaneously with a unit volume; (iii) buy-sell spreads are time constants unique to traders with distribution $\rho(L)$. The trader dynamics are then characterized by the midprice $z_{i}\equiv(b_{i}+a_{i})/2$; and (iv) trend-following random walks are assumed in the microscopic dynamics (Fig. 4a–c),

[TABLE]

with strength for trend following $c$, previous price movement $\Delta p$, and white Gaussian noise $\sigma\eta_{i}^{\mathrm{R}}$ with variance $\sigma^{2}$. Here, $c$, $\Delta p^{*}$, and $\sigma$ are assumed shared for all traders for simplicity. In this model, HFTs frequently modify their quoted price by successive submission and cancellation. Indeed, this model can be reformulated as a Poisson price modification process with high cancellation rate (see Appendix B.2). After transaction $a_{j}(t)=b_{i}(t)$ (Fig. 4b), the updated market price and its corresponding movement are recorded as

[TABLE]

and a requotation jump occurs (Fig. 4c),

[TABLE]

Here, $t+0$ implies the time after transaction. A unique character of this model is the order-book collective motion due to trend following (Fig. 4d). For $\Delta p>0$, the bid (ask) volume change tends to be positive (negative) near the best price (Fig. 4e), consistently with the layered order-book structure Yura2014 .

Kinetic formulation.– We next present an analytical solution to this model (2) according to kinetic theory Chapman1970 ; VanKampen . Let us first introduce the relative distance $r_{i}\equiv z_{i}-z_{{\mathrm{c.m.}}}$ from the “center of mass” $z_{{\mathrm{c.m.}}}\equiv\sum_{i}z_{i}/N$ (Fig. 4a), where the trend-following effect in Eq. (2) is absorbed into the dynamics of $z_{{\mathrm{c.m.}}}$. The dynamics of $r_{i}$ become simpler because trend-following effects disappear in this moving frame (see Appendix B.3). We next introduce the one-body (two-body) probability distribution as $\phi_{L}(r)$ ($\phi_{LL^{\prime}}(r,r^{\prime})$) conditional on traders’ buy-sell spreads. From the microscopic model (2), the lowest-order hierarchy equation is derived as $\partial\phi_{L}/\partial t\!=\!(\sigma^{2}/2)(\partial^{2}\phi_{L}/\partial r^{2})\!+\!N\!\sum_{s=\pm 1}\!\int\!dL^{\prime}\rho(L^{\prime})[J^{s}_{LL^{\prime}}(r\!+\!sL/2)\!-\!J^{s}_{LL^{\prime}}]$ with J_{LL^{\prime}}^{s}(r)\!\equiv\!(\sigma^{2}/2)|\tilde{\partial}_{rr^{\prime}}|\phi_{LL^{\prime}}(r,r^{\prime})\big{|}_{r\!-\!r^{\prime}\!=s(L\!+\!L^{\prime})/2} and $|\tilde{\partial}_{rr^{\prime}}|f\equiv|\partial f/\partial r|+|\partial f/\partial r^{\prime}|$ (see Appendix B.5). By assuming “molecular chaos”

[TABLE]

we derive the Boltzmann-like equation with collision integrals for the order book:

[TABLE]

with \tilde{J}_{LL^{\prime}}^{s}(r)\!\equiv\!(\sigma^{2}/2)|\tilde{\partial}_{rr^{\prime}}|\{\phi_{L}(r)\phi_{L^{\prime}}(r^{\prime})\}\big{|}_{r\!-\!r^{\prime}\!=s(L\!+\!L^{\prime})/2}. Here, $s=+1$ ($s=-1$) represents transactions as bidder (asker). Because traders exhibit collective motion arising from trend following, a Langevin-like equation is also derived as the macroscopic description of the model (2),

[TABLE]

where $\tau(T)$ and $\zeta(T)$ are transaction time interval and random noise at the $T$th tick time, respectively. The first trend-following term corresponds to the momentum inertia in the conventional Langevin equation.

Equations (6) and (7) can be analytically assessed for $N\to\infty$. We first set the buy-sell spread distribution as

[TABLE]

with decay length $L^{*}=15.5\pm 0.2$ tpip, empirically validated in our data set (Fig. 5a). The solution to Eq. (6) for $N\to\infty$ is given by $\phi_{L}(r)=(4/L^{2})\max\{L/2-|r|,0\}$. The average order-book profile $f_{\mathrm{A}}(r)=\int dL\rho(L)\phi_{L}(r-L/2)$ is then given for $r>0$ by

[TABLE]

The statistics of $\tau(T)$ in the macroscopic model (7) is derived from the mesoscopic model (6), and the tail of the price movement is approximately given by

[TABLE]

with decay length $\kappa\approx 2\Delta z^{*}/3$, average movement from trend following $\Delta z^{*}\equiv c\tau^{*}$, average transaction interval $\tau^{*}\approx 3L^{*2}/N\sigma^{2}$, and complementary cumulative distribution function (CDF) $P(\geq|\Delta p|;\kappa)$ (see also Appendices B.6 and B.7 for numerical validation).

Mesoscopic and macroscopic data analysis.– We next investigated whether our microscopic model is consistent with our data set. The empirical daily profile was first studied for the average ask order book for the best prices of HFTs $f_{\mathrm{A}}(r)$ (Fig. 5b). Surprisingly, we found a quantitative agreement with our theory (9) without any fitting parameters, which strongly supports the validity of our description.

The two-hourly segmented CDF for the price movement is also evaluated in one-tick precision $P^{\mathrm{2h}}(\geq|\Delta p|;\kappa)$ (Fig. 5c), which obeys an exponential law that is qualitatively consistent with our theoretical prediction (10). The value of the two-hourly decay length $\kappa$ fluctuates significantly during a week. To remove this nonstationary feature, we introduced the two-hourly scaled CDF $\tilde{P}^{\mathrm{2h}}(\geq|\Delta\tilde{p}|)\equiv P^{\mathrm{2h}}(\geq\kappa|\Delta\tilde{p}|;\kappa)/Z$ with scaling parameters $\kappa$ and $Z$ (Fig. 5d), thereby incorporating the two-hourly exponential law for the whole week.

The price movements obey an exponential law for short periods but simultaneously obey a power law over long periods with exponent $\alpha=3.6\pm 0.13$ (Fig. 5e). This apparent discrepancy originates from the power-law nature of the decay length $\kappa$. Because $\kappa$ approximately obeys a power-law CDF $Q(\geq\kappa)\sim\kappa^{-m}$ over the week with $m=3.5\pm 0.13$ (Fig. 5f), the one-week CDF $P^{\mathrm{w}}(\geq|\Delta p|)$ asymptotically obeys the power law as a superposition of the two-hourly segmented exponential CDF,

[TABLE]

with $Q(\kappa)\equiv-dQ(\geq\kappa)/d\kappa$, consistently with empirical exponent $\alpha\approx m$. Our result is therefore consistent with the previous reported power law Plerou1999 ; Lux1996 ; Guillaume1997 ; Longin1996 as a nonstationary property of $\kappa$.

Since our trend-following HFT model exhibits the order-book collective motion (Fig. 4d and e), this model can reproduce the layered order-book structure Yura2014 (see Appendix B.8). Let us define $c_{r}^{-}(c_{r}^{+})$ and $a_{r}^{-}(a_{r}^{+})$ as the number of bid (ask) submission and cancellation between one tick at the relative distance $r$ from the market midprice. We also define the number change $N_{r}^{-}=c_{r}^{-}-a_{r}^{-}$ $(N_{r}^{+}=c_{r}^{+}-a_{r}^{+})$ at the distance $r$ for the bid (ask) side. Correlation coefficient $C_{r}^{-}(C_{r}^{+})$ is plotted in Fig. 5g between $N_{r}^{-}(N_{r}^{+})$ and $\Delta p$, showing positive and negative correlation in the inner (outer) and outer (inner) layers, respectively. We further show a linear correlation between the price movement $\Delta p$ and the total number change in the inner layer $N_{\mathrm{inner}}\equiv\int_{-\infty}^{\gamma_{\mathrm{c}}}dr(N_{r}^{-}-N_{r}^{+})$. The trend-following HFT model is thus qualitatively consistent with the previous findings Yura2014 (see Appendix B.9 for data analyses), implying that the layered structure was the direct consequence of the collective motion.

Discussion. – We have empirically studied the trend following of HFTs, inducing the collective motion of the order book. This property has not been captured in the previous order-book model Slanina2014 ; Maslov2000 ; Daniels2003 ; Smith2003 ; Bouchaud2002 ; Farmer2005 and was critical in reproducing our empirical findings. Indeed, none of our empirical findings, the order-book profile, the exponential price movement, and the layered order-book structure Yura2014 were reproduced by the previous order-book model under realistic parameters in the absence of the collective motion (see Appendix B.10). We expect that introduction of this collective motion to order-book models would be the key to replicate these empirical findings.

Conclusion.– We have established both a microscopic model and a kinetic theory for FX traders by direct observation of the HFTs’ dynamics, quantitatively agreeing with empirical results under minimal assumptions. In the stream of econophysics, our model (2) is the first microscopic model directly supported by microscopic dynamical evidence and exhibiting agreement with mesoscopic and macroscopic findings. We expect that a new stream arises toward systematic descriptions of the financial market based on microscopic evidence. Interested readers are referred to Ref. KanazawaFull for more mathematical details.

Appendix A Data Analysis

A.1 Market rule

We analyzed high-frequency trading data in Electronic Broking Services (EBS), one of the biggest financial markets in the world. This market is continuously open except for weekends under few regulations. All trader activities were recorded for our data set with anonymized trader IDs and with one-millisecond time-precision from the 5th 18:00 to the 10th 22:00 GMT June 2016. The minimum price-precision was 0.005 yen for the USD/JPY pair at that time, and the currency unit used in this study is 0.001 yen, called the tenth pip (tpip). The minimum volume unit for transaction was one million USD, and the total monetary flow was about $68$ billion USD during this week. The EBS market is a hybrid market combining both quote-driven and order-driven systems, where traders have three options: limit order, market order, and cancellation. A limit order is an order quoting price with a certain volume and the quoted price displayed on the order book. A market order is an order to buy or sell currencies immediately at the available best price.

Here we define terminology in this paper. The highest bid and lowest ask quoted prices are called the market best bid and ask prices (denoted by $b_{\mathrm{M}}$ and $a_{\mathrm{M}}$), respectively (see Fig. 6a). The average of the market bid and ask prices is called the market midprice (denoted by $z_{\mathrm{M}}$). Also, the market transacted price $p$ (or the market price for short) means the price at which a transaction occurs in the market.

We note a central trading rule regarding the mutual credit lines between traders EBSRule . All market participants are required to set credit lines to counterparties in advance, and they cannot transact with each other in the absence of mutual credit. Therefore, traders sometimes transact at the worse price than the best market price.

A.2 Definition of the high frequency traders

For this paper, a high frequency trader (HFT) is defined as a trader who submits more than 500 times a day on average (i.e., more than 2500 times for the week). This definition is similar to that introduced in Ref. EBS_Schmit . As a few traders are unwilling to transact and often interrupt orders at the instant of submission, we excluded traders with live orders of less than $0.5\%$ of the transaction time. With this definition, the number of HFTs was 134 during this week, whereas the total number of traders was 1015. We note that the total number of traders who submitted limit orders was 922; the other 93 traders submitted only market orders. We also note that the presence of HFTs has rapidly grown recently and $87.8\%$ of the total orders were submitted by the HFTs in our data set.

Here we note a regulation on cancellations in this market, which is related to motivating HFTs to play the role of key liquidity providers (KLPs) EBSRule . For market stability, all traders are required not to cancel orders frequently; there is a threshold on the ratio between dealt quote and total number of quotations called the quote fill ratio (QFR). If the QFR of a trader is lower than a threshold, penalties are imposed on the trader in this market. However, there is a special rule to lower the threshold. If a trader maintains two-sided quotes continuously for a fixed time interval (called key liquidity hours), the trader qualifies as a KLP and is subject to a lower threshold QFR. Because HFTs tend to cancel orders frequently, they are typically KLPs as illustrated in Fig. 2a–c.

We also note the typical number of HFTs related to snapshots of the order book. We took snapshots of the order book after every transaction and counted the total number of different trader identifiers (IDs) for both bid and ask sides. The counting weight for an HFT quoting both sides is 1 and that for an HFT quoting one side is 1/2. We then plotted the average of the number of trader IDs for both bid and ask sides every two hours in Fig. 6b, showing the periodic intraday activity pattern of HFTs (i.e., $N$ tends to be small during 20:00-22:00 GMT). The typical number of HFT was about $35$ in our data set with this definition. The number of total volumes quoted by HFTs is typically about $80$. Admittedly, there is room for debate on which number is appropriate for the calibration of the total number of traders in our model; it remains a topic for future study.

A.3 Percentage of two-sided quotes

We calculated the percentage of two-sided quotes as follows; when a bid (ask) order is submitted by a trader, we check whether the corresponding ask (bid) orders exist. We then count the number of two-sided quotes for all traders at the submission of every order and finally divide it by the total number of submissions.

A.4 Cancellation ratio for individual traders

For each trader, we calculated the total number of canceled volumes over that of submitted volumes for the cancellation ratio of the trader. The cancellation ratio for the first, second, and third top HFTs were 98.59%, 99.93%, and 98.70%, respectively (or equivalently, their QFR were 1.41%, 0.07%, and 1.30%, respectively). The total cancellation ratio among all the HFTs was 94.42% (or equivalently the total QFR was 5.58%).

A.5 Buy-sell spread

The difference in the best bid and ask prices was studied as a buy-sell spread for an HFT. Samples where only both bid and ask prices exist are taken at one-second time-intervals for the insets in Fig. 2a–c and Fig. 5a. We plotted standard deviations of the averages as error bars for each point.

A.6 Trend-following effect

We explain the precise definition of the bid (ask) price of individual HFTs for the analysis of trend following. If a trader quotes both single-bid and single-ask orders at any time, the bid and ask prices are defined literally. In the presence of multiple bid or ask orders, we use the best value for the bid or ask orders as $b_{i}$ or $a_{i}$. In the absence of any bid or ask or both orders, we use the most recent bid or ask price as $b_{i}$ or $a_{i}$ for interpolation.

Because of the discrete nature of this data analysis, the probability that traders do not move at all (i.e., $\Delta z_{i}=0$) is estimated high. We therefore excluded the samples during an inactive time interval $\Delta z_{i}=0$ for the calculation of representative values in the following. This exception handling does not have a big impact on the hyperbolic structure in Eq. (1). Exceptional samples for which the bid or ask price is far from the market price by $0.1$ yen (0.02% of the total) are also excluded from the calculation of the conditional ensemble average $\langle\dots\rangle_{\Delta p}$. In Fig. 3a, data points are plotted whose samples are over 100 in each bin. The standard deviations of the conditional averages are plotted for each point as error bars. Also, median values in the top 20 HFTs are given using $c_{i}\sim 6.0$ tpip/tick and $\Delta p_{i}^{*}\sim 7.5$ tpip, which are estimated by the least squares methods implemented in gnuplot.

We have also calculated the standard deviation of quoted price movements for individual traders at one-tick precision in Fig. 3b. For the $i$th top HFT, we calculated the conditional variance $V_{\Delta p}[\Delta z_{i}]\equiv\langle(\Delta z_{i}-\langle\Delta z_{i}\rangle_{\Delta p})^{2}\rangle_{\Delta p}$ and took its square root. As can be seen from Fig. 3b, the standard deviation is approximately independent of $\Delta p$ for the top 20 HFTs. We note that the median value was $\sigma_{i}\sim 14.5$ tpip/tick. This observation is consistent with the assumption that only the drift term depends on $\Delta p$ but the random noise effect does not depend on $\Delta p$ in our microscopic model.

A.7 Average order-book profile

The daily average order-book profile is calculated for the best prices of the HFTs. We took snapshots of the order book for the best prices of the HFTs every second and we calculated its ensemble average every day. We also plotted standard deviations of the averages as error bars for each point.

A.8 Price movement distributions and decay length

The two-hourly segmented complementary cumulative distribution functions (CDFs) for the price movement $\Delta p$ are calculated in one-tick precision in Figs. 5c, d: $\Delta p(T)\equiv p(T+1)-p(T)$ with market price $p(T)$ at tick time $T$. The decay length $\kappa$ and its error were estimated by the least squares methods implemented in gnuplot (Figs. 5d, f) and the two-hourly scaled CDFs were plotted in Fig. 5d with the maximum samples excluded as outliers. The time-series of the estimated decay length $\kappa$ is plotted in Fig. 6b, showing that $\kappa$ was the longest just after the opening of the EBS market (the 5th 18:00–20:00 GMT). We conjectured that the decay length $\kappa$ was related to the market activity, represented by such as the number of HFTs during the time region. Indeed, the number of HFTs was also the least during the 5th 18:00–20:00 GMT in the week.

Appendix B Theoretical Analysis

B.1 Model dynamics

We explained the model dynamics as trend following random walks (2) with jump rules (3) and (4). These dynamics can be represented within the framework of Markovian stochastic processes using the $\delta$-functions. The stochastic dynamics can be written as

[TABLE]

where we have used the Itô convention. Here, $\tau_{k;ij}$ is the $k$-th collision time; jump size $\Delta z_{ij}$ between traders $i$ and $j$, post-collisional price $p^{{\mathrm{post}}}$, and price movement $\Delta p^{{\mathrm{post}}}$ are defined by

[TABLE]

with signature function $\mathrm{sgn}(x)$ defined by $\mathrm{sgn}(x)=x/|x|$ for $x\neq 0$ and $\mathrm{sgn}(0)=0$. Remarkably, the jump rule Eq. (13) corresponds to the contact condition and momentum exchange in the conventional kinetic theory. In the following, we present effective descriptions of this model for mesoscopic and macroscopic hierarchies.

B.2 Note on a Poisson price modification process

Since the Gaussian noise can be obtained by taking the high-frequent small jump limit for Poisson noises GardinerB , the model (12) can be reformulated as a Poisson price modification process with high-frequent cancellation rate. Here, let us focus on the quoted price dynamics for HFTs in the absence of transactions. As shown in Fig. 2a–c, HFTs tend to frequently and continuously modify their price by successive order cancellation and submission, possibly due to the market rule (i.e., they are required to maintain the continuous two-sided quote for a fixed time interval EBSRule ). On the basis of these characters, we can consider a Poisson cancellation model corresponding to the model (12). Let us introduce the order cancellation rate $\lambda$, which gives the cancellation probability during $[t,t+dt]$ as $\lambda dt$. The mean-cancellation interval is characterized by $\Delta t_{\mathrm{can}}\equiv 1/\lambda$. After cancellation, we assume that HFTs instantaneously requote their price to maintain continuous limit orders. In the absence of transaction, the requoted price is assumed to be described by a discrete version of Eq. (12) as

[TABLE]

with a standard Gaussian random number $\eta_{i}^{\mathrm{R}}$ according to our empirical finding (1). Transaction rule is also assumed the same as the continuous model (12). Here, the infinitesimal time step $dt$ is different from the mean-cancellation interval $\Delta t_{\mathrm{can}}$. A schematic trajectory described by this Poisson dynamics is illustrated in Fig. 7. The continuous model (12) is obtained in the high-frequent cancellation limit $\lambda\to\infty$ for the discrete model (14). The HFTs’ nature on high-frequent price modifications is thus reflected in the continuous model (12).

B.3 Introduction of the center of mass and the corresponding relative price

We here introduce the center of mass (c.m.) and the corresponding relative price (see Fig. 8 for a schematic):

[TABLE]

The dynamics of the c.m. and the relative price is given by

[TABLE]

Remarkably, trend following only appears in the dynamics of the c.m., but does not appear in that of the relative price. This is natural because trend following induces a collective behavior of traders, and can be absorbed into the dynamics of the c.m.. Furthermore, the contribution of $\xi$ is much smaller than that of $\sigma\eta^{\mathrm{R}}_{i}$ and $\eta_{i}^{\mathrm{T}}$ for $N\to\infty$: $|\xi|\ll|\sigma\eta_{i}^{\mathrm{R}}+\eta_{i}^{\mathrm{T}}|$. In the moving frame of the c.m., the dynamics of the relative price $r_{i}$ is thus simplified and approximately obeys the following dynamical equation:

[TABLE]

B.4 BBGKY Hierarchical equation for two-body problem: $N=2$

Before deriving the Bogoliubov–Born–Green–Kirkwood–Yvon (BBGKY) hierarchical equation for $N\gg 1$, we first consider the two-body system of traders to specify the collision integrals. Extension to the many-body problem will be studied in the next subsection. Let us denote the relative midprices of the first and second traders by $r_{1}$ and $r_{2}$ with constant spreads $L_{1}$ and $L_{2}$. The dynamics is given by

[TABLE]

with jump sizes $\Delta r_{1}$, $\Delta r_{2}$ and $k$-th transaction time $\tau_{k}$. Here, $\eta_{i;\varepsilon}$ is the colored Gaussian noise satisfying $\langle\eta_{i;\varepsilon}^{\mathrm{R}}(t)\eta_{j;\varepsilon}^{\mathrm{R}}(s)\rangle=\delta_{ij}e^{-|t-s|/\varepsilon}/2\varepsilon$ for $i,j=1,2$. Later, we shall take the $\varepsilon\to 0$ limit, whereby colored Gaussian noise $\eta_{i;\varepsilon}^{\mathrm{R}}$ converges to white Gaussian noise as $\lim_{\varepsilon\to 0}\langle\eta_{i;\varepsilon}^{\mathrm{R}}(t)\eta_{j;\varepsilon}^{\mathrm{R}}(s)\rangle=\delta_{ij}\delta(t-s)$. The $k$-th transaction time $\tau_{k}$ and the jump sizes $\Delta r_{1}$, $\Delta r_{2}$ are determined using the collision rule,

[TABLE]

We first derive the master equation for this system. For the two-body probability distribution function (PDF) $P_{12}(r_{1},r_{2})$, we exactly obtain a time-evolution equation

[TABLE]

where $|\tilde{\partial}_{12}|g(r_{1},r_{2})\equiv|\partial g(r_{1},r_{2})/\partial r_{1}|+|\partial g(r_{1},r_{2})/\partial r_{2}|$ is the sum of the absolute value of the partial derivatives for arbitrary $g(r_{1},r_{2})$. This equation can be derived as follows. For an arbitrary function $f(r_{1},r_{2})$, we obtain an identity

[TABLE]

where we have used the expansion of the $\delta$-function: $\delta(g(t))=\sum_{k=0}^{\infty}\delta(t-\tau_{k})/|g^{\prime}(\tau_{k})|$ with the $k$-th zero points, such that $g(\tau_{k})=0$ and $\tau_{k}<\tau_{k+1}$. Here we consider the direction of the collision; that is, $\eta_{1;\varepsilon}-\eta_{2;\varepsilon}$ must be positive just before the collision $r_{1}-r_{2}=(L_{1}+L_{2})/2$. Inversely, $\eta_{1;\varepsilon}-\eta_{2;\varepsilon}$ must be negative just before the collision $r_{1}-r_{2}=-(L_{1}+L_{2})/2$. We thus obtain

[TABLE]

We take the ensemble average of both sides to obtain

[TABLE]

Here the two-body PDF $P_{12}(x_{1},x_{2})$ characterizes the probability of $r_{1}\in[x_{1},x_{1}+dx_{1}]$ and $r_{2}\in[x_{2},x_{2}+dx_{2}]$ as $P_{12}(x_{1},x_{2})dx_{1}dx_{2}$. By substituting $f(r_{1},r_{2})=\delta(r_{1}-x_{1})\delta(r_{1}-x_{2})$, we obtain the master equation

[TABLE]

where the abbreviation symbol involving the derivatives is defined as $\tilde{\partial}_{12}\equiv\partial/\partial x_{1}-\partial/\partial x_{2}$, using the Novikov’s theorem Novikov for an arbitrary function $g(r_{1},r_{2})$ as

[TABLE]

We note that $\tilde{\partial}_{12}$ is a slightly different symbol from $|\tilde{\partial}_{12}|$ in terms of signatures (see Eq. (26) for their relation). We comment on the signature of the derivatives. Considering that $P_{12}(x_{1},x_{2})\geq 0$ for all $x_{1},x_{2}$ and $P_{12}(x_{1},x_{2})=0$ for $x_{1}-x_{2}>(L_{1}+L_{2})/2$, we obtain $(\partial P_{12}(x_{1},x_{2})/\partial x_{1})|_{x_{1}-x_{2}=(L_{1}+L_{2})/2}\leq 0$ and $(\partial P_{12}(x_{1},x_{2})/\partial x_{2})|_{x_{1}-x_{2}=(L_{1}+L_{2})/2}\geq 0$. We also obtain $(\partial P_{12}(x_{1},x_{2})/\partial x_{1})|_{x_{1}-x_{2}=-(L_{1}+L_{2})/2}\geq 0$ and $(\partial P_{12}(x_{1},x_{2})/\partial x_{2})|_{x_{1}-x_{2}=-(L_{1}+L_{2})/2}\leq 0$. In summary, we have

[TABLE]

By a change of notation $x_{1}\to r_{1}$ and $x_{2}\to r_{2}$, we obtain Eq. (20).

By integrating over $r_{2}$ on both sides, we obtain a hierarchical equation for the one-body PDF $P_{1}(r_{1})\equiv\int dr_{2}P_{12}(r_{1},r_{2})$ as

[TABLE]

where $J_{12}^{s}(r_{1})$ is the transaction probability per unit time as bidder $(s=+1)$ or asker $(s=-1)$. The first and second terms on the right-hand side account for the self-diffusion and collision terms, respectively. This is a lowest-order BBGKY hierarchical equation for the special case of $N=2$. Remarkably, the collision term has a quite similar mathematical structure to the collision integral in the conventional Boltzmann equation.

B.5 BBGKY hierarchical equation for many-body problem: $N\gg 1$

We have derived the hierarchical equation for the one-body PDF for the special case $N=2$. Here we extend the hierarchical equation for the many-body problem with $N\gg 1$. We first assume that the number of traders $N$ is sufficiently large that the spread distribution $\rho(L)$ can be approximated as a continuous function. The one-body and two-body PDFs conditional on buy-sell spread $L$ and $L^{\prime}$ are denoted by $\phi_{L}(r)$ and $\phi_{LL^{\prime}}(r,r^{\prime})$, respectively. We note the relations $P_{i}(r_{i})=\phi_{L_{i}}(r_{i})$ and $P_{ij}(r_{i},r_{j})=\phi_{L_{i}L_{j}}(r_{i},r_{j})$ hold for the one-body and two-body PDFs $P_{i}(r_{i})$ and $P_{ij}(r_{i},r_{j})$ for the traders $i$ and $j$, considering the symmetry between traders. Within the spirit of the Boltzmann equation, the dynamical equation for the one-body distribution $\phi_{L}(r)$ can be decomposed into two parts:

[TABLE]

with the self-diffusion term $(\sigma^{2}/2)(\partial^{2}\phi_{L}/\partial r^{2})$ and the collision integral $C(\phi_{LL^{\prime}})$. By extending the collision term in Eq. (27) for large $N\gg 1$, we can specify the collision integral as

[TABLE]

with the collision probability per unit time as bidder ($s=+1$) or asker ($s=-1$) against a trader with spread $L^{\prime}$. This is the Boltzmann-like equation, Eq. (6). We note that this BBGKY hierarchical equation can be systematically derived via the pseudo-Liouville equation. The derivation will be given in another technical paper in preparation KanazawaFull .

B.6 Boltzmann-like equation for finance

We next derive a closed equation for the one-body distribution function $\phi_{L}$ by assuming a mean-field approximation. Let us truncate the two-body correlation (i.e., molecular chaos in kinetic theory),

[TABLE]

A closed mean-field equation for the one-body distribution $\phi_{L}$ is thereby obtained,

[TABLE]

with the mean-field collision probability per unit time as bidder $(s=+1)$ or asker ($s=-1$)

[TABLE]

Equation (31) is a closed equation for the one-body distribution function, and corresponds to the Boltzmann equation in molecular kinetic theory.

Equation (31) can be analytically solved for $N\to\infty$, and the steady solution $\psi_{L}(r)$ is given by the tent function,

[TABLE]

Here, a technicality on the appropriate boundary condition will be summarized in another technical paper in preparation KanazawaFull . Note that the tent function (33) for the traders’ midprice order book implies the tent functions for both bid and ask order books in shifted coordinates (see Fig. 9 for a schematic). The average order-book profile for the ask side $f_{\mathrm{A}}(r)$ is then given by convolution with the tent function,

[TABLE]

We discuss here the intuitive meaning of the mean-field solution (34). The mean-field solution (34) is exactly zero at $r=\pm L/2$ as $\psi_{L}(+L/2)=\psi_{L}(-L/2)=0$, implying that the edge points $r=\pm L/2$ effectively play the role of hopping barriers at which the particle hops into $r=0$. Indeed, Eq. (33) gives exactly the same solution to the problem of the Brownian motion confined by hopping barriers, as shown in Sec. C.2. This is a reasonable result for the $N\to\infty$ limit, where the market is sufficiently liquid and most of the transactions occur just around $r=\pm L/2$.

B.6.1 Average order-book profile from the center of mass

If the spreads are distributed in accordance with the $\gamma$-distribution, as empirically studied in the main text, the average order-book profile is given by

[TABLE]

To check the validity of this formula, we performed Monte Carlo simulations of the microscopic model (12) (Fig. 10a), where the theoretical formula (35) works for various $N$. In the figure, we denote the relative price by $r_{{\mathrm{c.m.}}}$ to stress that it is defined from the c.m. as $r_{{\mathrm{c.m.}}}\equiv z-z_{{\mathrm{c.m.}}}$.

B.6.2 Average order-book profile from the market midprice

Technically, we have studied the average order-book profile $f_{\mathrm{A}}^{{\mathrm{c.m.}}}(r_{{\mathrm{c.m.}}})$ from the c.m. instead of that from the market midprice $f_{\mathrm{A}}^{\mathrm{mid}}(r_{\mathrm{mid}})$, because $f_{\mathrm{A}}^{{\mathrm{c.m.}}}(r_{{\mathrm{c.m.}}})$ is theoretically more tractable than $f_{\mathrm{A}}^{\mathrm{mid}}(r_{\mathrm{mid}})$. Here $r_{\mathrm{mid}}\equiv a_{i}-z_{\mathrm{M}}$ is the relative distance from the market midprice $z_{\mathrm{M}}$ for the ask price $a_{i}$ of the $i$th trader. Fortunately, they are asymptotically equivalent for the large $N$ limit and the above formulation is sufficient in understanding the average order-book $f_{\mathrm{A}}^{\mathrm{mid}}(r_{\mathrm{mid}})$ from the market midprice:

[TABLE]

To validate this asymptotic equivalence, we numerically demonstrate the average order-book profile $f_{\mathrm{A}}^{\mathrm{mid}}(r_{\mathrm{mid}})$ from the market midprice $z_{\mathrm{mid}}$ in Fig. 10b. This figure numerically shows that the average order-book formula (35) is valid even for the order-book from the market midprice.

B.6.3 Statistics of transaction interval

We comment on the statistics of the transaction interval $\tau$. In the mean-field approximation, the average of the transaction interval is given by

[TABLE]

which is phenomenologically derived in Sec. C.1 and is numerically validated in Fig. 10c. Note that this formula can be derived from the pseudo-Liouville equation more systematically KanazawaFull . Based on the average transaction interval (37), the CDF for the transaction interval $P(\geq\tau)$ is approximately given by the phenomenological formula,

[TABLE]

with transaction interval PDF $P(\tau)$. This formula is derived in Sec. C.2 and is numerically validated in Fig. 10d.

B.7 Langevin-like equation for finance

Here we derive phenomenologically the fundamental equation for the financial Brownian motion, which corresponds to the conventional Langevin equation. Let us denote the $T$th transaction price by $p(T)$ and the $T$th price movement by $\Delta p(T)\equiv p(T+1)-p(T)$. Here we focus on the effects of trend following obtained from Eq. (16), which induces inertia-like collective motion of the macroscopic dynamics. The dynamical equation for the price movement is thus given by

[TABLE]

where $\tau(T)$ is the time interval between the $T$th and $(T+1)$th transaction. The first and second terms originate from the trend following and random noise, respectively. Note that the statistics of $\tau(T)$ is derived from the mesoscopic model (31) as Eqs. (37) and (38).

Equation (39) governs the macroscopic dynamics of the financial Brownian motion, corresponding to the conventional Langevin equation. For small trend $|\Delta p|\ll\Delta p^{*}$, we indeed obtain a formal expression similar to the conventional Langevin equation,

[TABLE]

with $\tilde{\gamma}(T)\equiv(1-c\tau(T))/\Delta T$, $\tilde{\zeta}(T)\equiv\zeta(T)/(\Delta T)^{2}$, and $\Delta^{2}p(T)\equiv\Delta p(T+1)-\Delta p(T)$.

We next study the price movement distribution using the financial Langevin equation (39), which is however a stochastic difference equation that cannot be solved exactly. Nonetheless, its qualitative behavior can be assessed approximately by making the following two assumptions.

1. (i)

The effect of trend following is sufficiently large compared with random noise: $|c^{*}\tau(T)|\gg|\zeta(T)|$. 2. (ii)

The average movement by trend following $\Delta z^{*}\equiv c\tau^{*}$ is much larger than the saturation threshold: $\Delta z^{*}\gg\Delta p^{*}$.

Under condition (i), the qualitative behavior is governed by the statistics of the transaction interval $\tau(T)$. Under condition (ii), furthermore, the hyperbolic function in Eq. (1) can be approximated for large fluctuations as $\tanh(\Delta p/\Delta p^{*})\approx\mathrm{sgn}(\Delta p)$ and the term $\zeta(T)$ is irrelevant for the tail. Based on Eq. (38) for the transaction interval $\tau$, the price distribution $P(\geq|\Delta p|)$ is approximately obtained for $|\Delta p|\to\infty$ as

[TABLE]

with an estimated decay length of $\kappa\approx 2\Delta z^{*}/3$. The validity of this formula was numerically checked in Fig. 11.

B.8 Numerical analysis of the layered order-book structure for the HFT model

Here we show the detailed analysis to study the layered order-book structure for the HFT model according to the method in Ref. Yura2014 . The numerical simulation was performed for the Poisson price modification process (14) under the parameter set of $L^{*}=15.5$ tpip, $\Delta p^{*}=3.65$ tpip, $\Delta z^{*}=4.56$ tpip, $\Delta t_{\mathrm{can}}\equiv 1/\lambda=4/\tau^{*}$, and $N=50$. At the instant of an order submission and cancellation for the bid (ask) side, with bin-width of $1$ tpip we measured the relative depth $r$ from the market midprice $z_{\mathrm{M}}$, defined as $r\equiv z_{\mathrm{M}}-b_{i}$ ($r\equiv a_{i}-z_{\mathrm{M}}$), and we incremented the numbers of $c_{r}^{-}$ $(c_{r}^{+})$ and $a_{r}^{-}$ $(a_{r}^{+})$ by one, respectively. We then accumulated their numbers between $T$ and $T+1$ tick to obtain one sample of $N_{r}^{-}(T)\equiv c_{r}^{-}(T)-a_{r}^{-}(T)$ $(N_{r}^{+}(T)\equiv c_{r}^{+}(T)-a_{r}^{+}(T))$. We also study the movement of the market price $\Delta p(T)\equiv p(T+1)-p(T)$ and calculated Pearson’s correlation coefficient $C_{r}^{-}$ $(C_{r}^{+})$ between $\Delta p(T)$ and $N_{r}^{-}(T)$ ($N_{r}^{+}(T)$) as shown in Fig. 5g. The crossover point was estimated to be $\gamma_{\mathrm{c}}\approx 16.5$ tpip in the numerical simulation.

We next study the linear correlation between the number change $N_{\mathrm{inner}}$ in the inner layer and the price movement $\Delta p$. Between $T$ and $T+1$ tick, we take a sample of both $N_{r}^{-}(T)$ and $N_{r}^{+}(T)$, and calculated their integral in the inner layer as $N_{\mathrm{inner}}(T)\equiv\int_{-\infty}^{\gamma_{\mathrm{c}}}dr\{N_{r}^{-}(T)-N_{r}^{+}(T)\}$. We then calculated the correlation between $N_{\mathrm{inner}}(T)$ and $\Delta p(T)$ for Fig. 5h. We have plotted the average of $\Delta p(T)$ conditional on $N_{\mathrm{inner}}(T)$ with errorbars representing the conditional variance. The figure shows their significant linear correlation of Pearson’s coefficient $0.63$.

B.9 Empirical analysis of the layered structure of the order book in the data set

We show the empirical layered structure of the HFTs’ order book in our data set. According to the essentially same method in Sec. B.8, we have calculated the layered structure as shown in Fig. 12a and b. The volume change in the inner layer $N_{\mathrm{inner}}(T)$ has a significant correlation of Pearson’s coefficient $0.616$ with the price movement $\Delta p(T)$.

For consistency throughout this Letter, we have focused on the best prices of HFTs for the correlation analysis in Fig. 12a and b. In other words, we incremented $c_{r}^{-}$ and $c_{r}^{+}$ when the newly quoted price was the best price of the trader. Also, we incremented $a_{r}^{-}$ and $a_{r}^{+}$ when the price of the canceled order was the best price of the trader.

B.10 Numerical comparison with the zero-intelligence order-book models

Here we compare our empirical findings with the zero-intelligence order-book (ZI-OB) model Daniels2003 ; Smith2003 ; Bouchaud2002 . The basic ZI-OB model is the uniform decomposition order-book model introduced in Ref. Daniels2003 ; Smith2003 , where both submission and cancellation are assumed to obey the homogeneous Poisson processes. To understand real average order-book profiles, in Ref. Bouchaud2002 , the uniform submission rate was replaced with a real nonuniform submission rate obeying a power law. Here we study the improved ZI-OB model in Ref. Bouchaud2002 from the viewpoint of the consistency with our empirical findings. The inputs to the ZI-OB model are the following three components.

Submission rate density $\mu(r_{\mathrm{mid}})$: limit order submissions are assumed to obey the inhomogeneous Poisson process characterized by the submission rate $\mu(r_{\mathrm{mid}})$ with relative depth $r_{\mathrm{mid}}$ from the market midprice. In other words, a new limit order is submitted in the range $[r_{\mathrm{mid}},r_{\mathrm{mid}}+dr_{\mathrm{mid}}]$ between time interval $[t,t+dt]$ with probability of

[TABLE]

The empirical submission histogram in our data set is depicted in Fig. 12c, showing a power-law tail with exponent $2.9$. For our numerical implementation, the limit order submission rate is directly fixed from the empirical submission histogram for $r_{\mathrm{mid}}\geq 0$. The total submission rate is given by

[TABLE]

characterizing the frequency of total submissions. The gender of order (i.e., buy or sell) is randomly selected with equal probability. 2. 2.

Cancellation rate $\lambda$: any order is assumed to be cancel according to the homogeneous Poisson process with intensity $\lambda$. In other words, an order is canceled between time interval $[t,t+dt]$ with probability of

[TABLE] 3. 3.

Market order rate $\omega$: market orders are assumed to obey the Poisson process with intensity $\omega$. In other words, a buy or sell market order is submitted between interval $[t,t+dt]$ with probability of

[TABLE]

The gender of order is randomly selected with equal probability.

These parameters characterize the order-book dynamics in the steady state. For example, the average total order-book volume $N_{\mathrm{vol}}$ in both sides and the QFR (i.e., the probability for an order to be transacted finally) are given by

[TABLE]

respectively. These relations are deduced from the conservation of order flux in the steady state: $\mu_{\mathrm{tot}}\approx\lambda N_{\mathrm{vol}}+\omega$.

B.10.1 Numerical simulation with realistic parameters

We first consider a numerical simulation based on realistic parameters in our data set with minimum price precision of $1$ tpip. The submission rate density $\mu(r_{\mathrm{mid}})$ is directly fixed from the empirical submission histogram in our data set (Fig. 12c). The cancellation and market order rates are fixed as $\lambda/\mu_{\mathrm{tot}}=9.5\times 10^{-3}$ and $\omega/\mu_{\mathrm{tot}}=0.05$ to satisfy $N_{\mathrm{vol}}\approx 100$ and $\mathrm{QFR}\approx 5\%$. Though these parameters were realistic in our data set, the numerical results in Fig. 13a–d were not consistent with the empirical findings. Indeed, the price movement in the ZI-OB model obeyed the Gaussian statistics (Fig. 13a), which is different from the empirical exponential law in our data set. The numerical average order-book profile did not quantitatively fit the real order-book profile (Fig. 13b). In addition, the layered structure of the order book did not emerge from the ZI-OB model (Fig. 13c and d). To replicate these empirical findings, in particular the layered order-book structure, we conjectured that the collective motion of the limit order book (i.e., the microscopic trend-following behavior) needs to be incorporated with conventional order-book models.

B.10.2 Numerical simulation with adjusted parameters

In Ref. Bouchaud2002 , the possibility of the ZI-OB model was studied to fit realistic order-book profiles by adjusting parameters. In the same way, we here seek the possibility to adjust the model parameters in replicating the real order-book profile in our data set. By fixing the average order-book volume as $N_{\mathrm{vol}}\approx 100$, we adjusted the market order rate $\omega$ as a fitting parameter to replicate the real order-book profile in our data set (see Fig. 13e–h). By inputting $\omega/\mu_{\mathrm{tot}}=0.75$ (or equivalently $\mathrm{QFR}\approx 75\%$), the ZI-OB model replicated the real order-book profile as shown in Fig. 13f. Instead, however, other numerical results of the ZI-OB model were not consistent with the exponential price movement statistics (Fig. 13e for the power-law price movement) nor the layered order-book structure (Fig. 13g and h). In addition, the parameter adjustment implied $\mathrm{QFR}\approx 75\%$, which was over ten-times larger than the real QFR (see Sec. A.4). We thus conclude that our empirical results were not consistently replicated by the ZI-OB model under realistic parameters, at least in our data set.

Appendix C Technical Issues for Derivation

C.1 Brownian motion confined by hopping barriers

In this subsection, we study the Brownian motion confined by the hopping barriers at $r=\pm L/2$ (see Fig. 14 for a schematic). Let us assume that a particle moves randomly in the absence of collision for $r\in(-L/2,L/2)$. We then place hopping barriers at $r=\pm L/2$, and we assume that the particle moves to the origin $r=0$ after collisions. The particle’s position $r(t)$ then obeys the dynamical equation

[TABLE]

where $\hat{\eta}^{\mathrm{R}}$ is the white Gaussian noise with unit variance, and $\hat{\eta}^{\mathrm{T}}_{+}$ and $\hat{\eta}^{\mathrm{T}}_{-}$ are respectively the jump terms originating from the hopping barriers at $r=\pm L/2$. Here, the $k$th collision times $\tau_{k}^{+}$ and $\tau_{k}^{-}$ at the barriers $r=\pm L/2$ satisfy the relation $r(\tau_{k}^{\pm})=\pm L/2$. In a parallel calculation to that in Sec. B.4, the dynamical equation for the probability distribution function $P(r)$ is given by

[TABLE]

The steady solution is then given by the tent function, $P_{\mathrm{SS}}(r)\equiv\lim_{t\to\infty}P(r)=\psi_{L}(r),$ which is the same as the mean-field solution (33) for $N\to\infty$. This implies that the mean-field description corresponds to Brownian motion confined by the hopping barriers in the limit $N\to\infty$.

From the above picture, we can derive Eq. (37) for the mean transaction interval asymptotically in terms of $N$. Because a single particle in our model behaves as the Brownian motion confined by the hopping barriers for $N\to\infty$, the mean transaction interval for a single particle can be derived by considering the survival rate problem for the model (47). According to Ref. Yamada2009 , the mean transaction interval is given by $L^{2}/4\sigma^{2}$ for a single particle. We next derive the mean transaction interval for the whole system. A count of the number of collisions $n_{L}$ for the spread $L$ during the time interval $T$ yields $n_{L}=T/(L^{2}/4\sigma^{2})$, when $T$ is sufficiently large. The total number of collisions $n_{\mathrm{tot}}$ is then given by

[TABLE]

where there are duplicate counts because any transaction occurs as a binary collision. Considering the duplicate counts, the mean transaction interval $\tau^{*}$ for the whole system is given by $\tau^{*}=T/(n_{\mathrm{tot}}/2)$, which implies Eq. (37).

C.2 Transaction interval distribution

The phenomenological estimation of the cumulative distribution for transaction interval (38) is presented here. Let us assume that the arrival-time intervals of a bidder and an asker at the center of mass obey the Poisson statistics:

[TABLE]

with the characteristic time interval $a$. $P_{\mathrm{A}}(\tau_{\mathrm{A}})$ ($P_{\mathrm{B}}(\tau_{\mathrm{B}})$) and $P_{\mathrm{A}}(\geq\tau_{\mathrm{A}})$ ($P_{\mathrm{B}}(\geq\tau_{\mathrm{B}})$) are the PDFs and CDFs of arrival time intervals for an asker (a bidder), respectively. We also assume that the transaction occurs when both bidder and asker arrive at the center of mass. This picture implies that the transaction interval $\tau$ is approximately given by

[TABLE]

where we have used a formula for the order statistics David2003 . Considering the consistency between Eq. (51) and the mean transaction interval (37), we obtain the self-consistent condition $\tau^{*}=3a/2$. Equation (38) then follows.