An intelligent financial portfolio trading strategy using deep Q-learning

TL;DR

This paper introduces a deep Q-learning based portfolio trading strategy that models the process as a Markov decision process with a discrete action space, enabling dynamic and practical multi-asset trading.

Contribution

It presents a novel approach that transforms infeasible actions into feasible ones, overcomes dimensionality issues, and simulates all feasible actions for improved multi-asset trading strategies.

Findings

Outperforms benchmark strategies in backtests

Handles infeasible actions with a mapping function

Effectively models multi-asset trading with deep Q-learning

Abstract

Portfolio traders strive to identify dynamic portfolio allocation schemes so that their total budgets are efficiently allocated through the investment horizon. This study proposes a novel portfolio trading strategy in which an intelligent agent is trained to identify an optimal trading action by using deep Q-learning. We formulate a Markov decision process model for the portfolio trading process, and the model adopts a discrete combinatorial action space, determining the trading direction at prespecified trading size for each asset, to ensure practical applicability. Our novel portfolio trading strategy takes advantage of three features to outperform in real-world trading. First, a mapping function is devised to handle and transform an initially found but infeasible action into a feasible action closest to the originally proposed ideal action. Second, by overcoming the dimensionality problem, this study establishes models of agent and Q-network for deriving a multi-asset trading strategy in the predefined action space. Last, this study introduces a technique that has the advantage of deriving a well-fitted multi-asset trading strategy by designing an agent to simulate all feasible actions in each state. To validate our approach, we conduct backtests for two representative portfolios and demonstrate superior results over the benchmark strategies.

Figures

Tables

Table 1. Table 1: Summary of notations

Decision variables
$a_t=(a_{t,1},a_{t,2},\dots ,a_{t,I})$	agent’s action at the end of period $t$ $\{a_t\in {\mathbb{Z}}^I:a_{t,i}\in \{-1,0,1\}\forall i\}$
Set and indices
$i=0,1,2,\dots ,I$	portfolio asset index (i=0 represents cash)
$t$	time period index
$S^{-}\left(a_t\right)$	set of an index of selling assets when an agent takes action $a_t$ (i.e., )
$S^{+}\left(a_t\right)$	set of an index of buying assets when an agent takes action $a_t$ (i.e., )
Parameters
$n$	size of the time window containing recent previous market features
$P_t$	portfolio value changed by the action at the end of period $t$
$P_t^{\prime }$	portfolio value before the agent takes an action at the end of period $t$
$P_t^s$	portfolio value at the end of period $t$ when the agent takes no action at the end of the previous period $t-1$ (static portfolio value in period $t$)
$w_{t,i}$	proportion of asset $i$ changed by the action at the end of period $t$
$w_{t,i}^{\prime }$	proportion of asset $i$ before the agent takes an action at the end of period $t$
${\hat{w}}_{t,i}^{\prime }$	auxiliary parameter used to derive $w_{t,i}$
$c_t$	decay rate of transaction costs at the end of period $t$
$c^{-}$	transaction cost rate for selling
$c^{+}$	transaction cost for buying
$\delta$	trading size for selling or buying
${\rho }_t$	return rate of the portfolio in period $t$ $\left(=\frac{P_t-P_{t-1}}{P_{t-1}}\right)$
$o_{t,i}$	opening price of asset $i$ in period $t$
$p_{t,i}$	closing price of asset $i$ in period $t$
$h_{t,i}$	highest price of asset $i$ in period $t$
$l_{t,i}$	lowest price of asset $i$ in period $t$
$v_{t,i}$	volume of asset $i$ in period $t$

Table 2. Table 2: Summary of market features

Features
$k_{t,i}^c$	rate of change of the closing price of asset $i$ in period $t$ $\left(=\frac{p_{t,i}-p_{t-1,i}}{p_{t-1,i}}\right)$
$k_{t,i}^o$	ratio of the opening price in period $t$ to the closing price in period $t-1$ for asset $i$ $\left(=\frac{o_{t,i}-p_{t-1,i}}{p_{t-1,i}}\right)$
$k_{t,i}^h$	ratio of the closing price to the highest price of asset $i$ in period $t$ $\left(=\frac{p_{t,i}-h_{t,i}}{h_{t,i}}\right)$
$k_{t,i}^l$	ratio of the closing price to the lowest price of asset $i$ in period $t$ $\left(=\frac{p_{t,i}-l_{t,i}}{l_{t,i}}\right)$
$k_{t,i}^v$	rate of change of the volume of asset $i$ in period $t$ $\left(=\frac{v_{t,i}-v_{t-1,i}}{v_{t-1,i}}\right)$

Table 3. Table 3: Test portfolios

Assets	Portfolio
	US Portfolio (US-ETF)	Korean Portfolio (KOR-IDX)
Asset 1	SPDR S&P 5001	KOSPI 100 index
Asset 2	iShares Russell 1000 Value2	Midcap KOSPI index
Asset 3	iShares Microcap3	Microcap KOSPI index

Table 4. Table 4: hyper-parameter summary

hyper-parameter	value	hyper-parameter	value
time window size $\left(n\right)$	20	replay memory size	2000
learning rate $\left(\alpha \right)$	1e-7	number of epochs	500
distribution parameter $\left(\beta \right)$	0.3	discount factor $\left(\gamma \right)$	0.9
DNN input dimension	64	batch size	32
DNN layer	2	encoder LSTM layer	1
DNN 1st layer dimension	64	hidden dim of encoder LSTM	128
DNN 2nd layer dimension	32	output dim of encoder LSTM	20

Table 5. Table 5: The number of changes in the phase of trading direction for DQN strategy without/with the mapping function in the two test portfolio cases

	US portfolio			Korean portfolio
mapping function	asset 1	asset 2	asset 3	asset 1	asset 2	asset 3
without	83	107	64	74	128	63
with	54	40	37	42	68	40
difference	-34.9%	-62.6%	-42.1%	-43.2%	-46.8%	-36.5%

Table 6. Table 6: Output performance measure values for our DQN strategy and the benchmark strategies in the two test portfolio cases

	US portfolio			Korean portfolio
strategy	$CR$	$SR$	$AT$	$CR$	$SR$	$AT$
	10.921%	1.308	0.000%	7.913%	0.890	0.000%
$RN$*	9.446%	1.143	1.029%	7.358%	0.381	1.032%
$MO$	10.372%	1.114	1.368%	8.773%	0.840	1.233%
$RV$	5.891%	0.642	1.404%	2.855%	0.147	1.370%
DQN	12.634%	1.382	0.954%	9.933%	0.946	0.989%

Equations

$$s_t=(X_t,w_t^{\prime })\text{, }$$

$$w_t^{\prime }=(w_{t,0}^{\prime },w_{t,1}^{\prime },w_{t,2}^{\prime },...,w_{t,I}^{\prime }{)}^T\text{, }$$

$$X_t=[K_t^c,K_t^o,K_t^h,K_t^l,K_t^v]\text{, }$$

$$k_t^x=(k_{t,1}^x,k_{t,2}^x,...,k_{t,I}^x{)}^T\forall x\in \{c,o,h,l,v\}\text{, }$$

$$K_t^x=[k_{t-n+1}^x|k_{t-n+2}^x|...|k_t^x]\forall x\in \{c,o,h,l,v\}\text{, }$$

$$w_t=(w_{t,0},w_{t,1},w_{t,2},...,w_{t,I}{)}^T\text{, }$$

$$w_t\cdot \vec{1}=w_t^{\prime }\cdot \vec{1}=1\forall t\text{, }$$

$$P_t^{\prime }=P_{t-1}{w}_{t-1}\cdot \varphi (k_t^c)\forall t\text{, }$$

$$w_t^{\prime }=\frac{w_{t-1}\odot \varphi (k_t^c)}{w_{t-1}\cdot \varphi (k_t^c)}\forall t\text{, }$$

$$\displaystyle c_{t}=\frac{\delta}{P_{t}^{\prime}}\biggl{(}c^{-}{\mid}S^{-}(a_{t}){\mid}+c^{+}{\mid}S^{+}(a_{t}){\mid}\biggr{)}\hskip 28.45274pt\forall t\text{, }$$

$$P_t=P_t^{\prime }(1-c_t)\forall t\text{, }$$

$${\hat{w}}_t^{\prime }=({\hat{w}}_{t,0}^{\prime },{\hat{w}}_{t,1}^{\prime },{\hat{w}}_{t,2}^{\prime },...,{\hat{w}}_{t,I}^{\prime }{)}^T\text{, }$$

$${\hat{w}}_{t,i}^{\prime }=\{\begin{array}{l}{w}_{t,i}^{\prime }-\frac{\delta }{P_t^{\prime }}ifi\in S^{-}(a_t),\\ w_{t,i}^{\prime }+\frac{\delta }{P_t^{\prime }}ifi\in S^{+}(a_t),\\ w_{t,i}^{\prime }otherwise\end{array}{\forall }_{i=1\dots I}\text{, }$$

$$\displaystyle\hat{w}_{t,0}^{\prime}=w_{t,0}^{\prime}+\frac{\delta}{P_{t}^{\prime}}\biggl{(}(1-c^{-}){\mid}S^{-}(a_{t}){\mid}-(1+c^{+}){\mid}S^{+}(a_{t}){\mid}\biggl{)}\text{, }$$

$$w_t=\frac{{\hat{w}}_t^{\prime }}{{\hat{w}}_t^{\prime }\cdot \overrightarrow{1}}\text{,}$$

$$r_t=\frac{P_{t+1}^{\prime }-P_{t+1}^s}{P_{t+1}^s}\text{, }$$

$$P_{t+1}^s=P_t^{\prime }{w}_t^{\prime }\cdot \varphi (k_{t+1}^c)\text{, }$$

$$g_{\beta }(y)=\frac{\beta (1-\beta {)}^{y_v-y-1}}{1-(1-\beta {)}^N}\text{, }$$

$$CR=\frac{P_{t_f}-P_0}{P_0}\times 100(\%)\text{, }$$

$$SR=\frac{E[{\rho }_t-{\rho }_f]}{std({\rho }_t)}\times \sqrt{252}\text{, }$$

$$AT=\frac{1}{2t_f}\sum _{t=0}^{t_f}\sum _{i=1}^I\mid {\hat{w}}_{t,i}^{\prime }-w_{t,i}^{\prime }\mid \times 100(\%)\text{. }$$

Full text

1 Introduction

A goal of financial portfolio trading is maximizing the trader’s monetary wealth by allocating capital to a basket of assets in a portfolio over the periods during the investment horizon. Thus, portfolio trading is the most important investment practice in the buy-side financial industry. Portfolio traders strive to establish trading strategies that can properly allocate capital to financial assets in response to time-varying market conditions. Typical objective functions for trading strategy optimization include expected returns and the Sharpe ratio (i.e., risk-adjusted returns). In addition to optimizing an objective function, a trading strategy should achieve a reasonable turnover rate so that it is applicable to real-world financial trading. If the turnover rate is not reasonable, transaction costs hurt overall trading performance.

Portfolio trading is an optimization problem that involves a sequential decision-making process across multiple rebalancing periods. In this process, the stochastic components of time-varying market variables should be considered. Thus, the problem of deriving an optimal portfolio trading strategy has traditionally been formulated as a stochastic optimization problem [10, 15, 19]. To handle these stochastic components over multiple periods, most related studies have developed heuristic methods [5, 7, 8, 13, 21, 29, 38, 39]. In very recent years, reinforcement learning (RL) has become another popular approach for financial portfolio trading problems. In RL methods, a learning agent can understand a complex financial environment by attempting various trading actions and revising its trading action policy, and then optimize their trading strategies based on these experiences. In addition, these methods have the important advantage that learning agents can update their trading strategies based on their experiences on future trading days. Instead of simply maintaining trading strategies derived from historical data, learning agents can adapt their strategies using their observed experiences on each real trading day [36]. With these advantages and the increasing popularity of RL algorithms, many previous studies have been conducted to apply RL algorithms to various portfolio trading problem settings [1, 2, 3, 6, 11, 12, 14, 17, 18, 24, 25, 26, 27, 28, 30, 38].

According to the recent evolution of RL methods, some researchers [12, 17, 18] have started to use deep RL (DRL) methods, which is the combination of RL and deep neural network (DNN), to overcome the unstable performance of previous RL methods. Nevertheless, we believe that this line of study is yet to mature in terms of practical applicability because of the following two reasons. First, most studies based on DRL methods focus on a single-asset trading [12, 17]. Because most traders generally have multiple securities, additional decision-making steps are necessary even though single asset trading rules are derived. Second, even if a study dealing with the multi-asset portfolio trading, the actions determine portfolio weights [18]. The action spaces that determine the portfolio weights cannot be followed directly because it requires additional decisions on how to satisfy the target weight. We will explain this issue in more detail in Section 2.3.

To overcome the current limitations, this study proposes a new approach for deriving a multi-asset portfolio trading strategy using deep Q-learning, one of the most popular DRL methods. In this study, we focus on a multi-asset trading strategy and define an intuitive trading action set that can be interpreted as direct investment guides to traders. In the action space used in this study, each action includes trading directions corresponding to each asset in a portfolio, and each trading direction comprises either holding each asset or buying or selling each asset at a prespecified trading size. Although a recent study [30] argues that optimizing a trading strategy based on a discrete action space has a negative effect, we find that our discrete action space modeling allows for a lower turnover rate and is more practical than continuous action space modeling is.

To develop a practical multi-asset trading strategy, this study tackles a few challenging aspects. First, setting a discrete combinatorial action space may lead to infeasible actions, and, thus, we may derive an unreasonable trading strategy (i.e., a strategy with frequent and pointless portfolio weight changes that only leads to more transaction costs). To address this issue, we introduce a mapping function that enables the agent to prevent the selection of unreasonable actions by mapping infeasible actions onto similar and valuable actions. By applying this mapping function, we can derive a reasonable trading strategy in the practical action space. Second, the action space that determines a trading direction for each asset in the portfolio has a dimensionality problem [24]. As the number of assets in portfolio increasing, then the size of action space increases exponentially because a trading agent must determine a combination of trading directions for several assets in the portfolio. This is why previous studies related to this action space have only considered a single asset or a single risky asset with a risk-free asset trading. Therefore, in this study, we overcome this limitation and conduct the first study of multi-asset trading in the practical action space by using DQL. Third, although we use years of financial data, these data may not provide enough training data for the DRL agent to learn a multi-asset trading strategy in the financial environment. Because learning a strategy mapping from a joint state to a joint action is necessary lots of data. There is a given amount of data, so we need to make the agent gains more experience within the training data and learns as much as possible. Thus, we achieve sufficient learning by simulating all feasible actions in each state and then updating the agent’s trading strategy using the learning experiences from the simulation results. This technique allows the agent to gain and learn enough experience to derive a well-fitted multi-asset trading strategy.

The rest of this paper is organized as follows. In Section 2, we first review the related literature and present the differences between our study and previous studies. Section 3 describes the definition of our problem, and Section 4 introduces our approach for deriving an intelligent trading strategy. In Section 5, we provide experimental results to validate the advantages of our approach. Finally, we conclude in Section 6 by providing relevant implications and identifying directions for future research.

2 Literature Review

Portfolio trading is an optimization problem that involves a sequential decision-making process over multiple rebalancing periods. In addition, the stochastic components of market variables should be considered in this process. Thus, traditionally, the derivation of portfolio trading strategies has been formulated as a stochastic programming problem to find an optimal trading strategy. Recently, much effort has been made to solve this stochastic optimization problem using a learning-based approach, RL. To formulate this stochastic optimization problem, it is necessary to determine how to measure the features of the stochastic components corresponding to changes in the financial market. Utilizing technical indicators is more common than utilizing the fundamental indexes of securities in daily frequency portfolio trading, as in our study.

This section reviews how previous studies have attempted to model stochastic market components to formulate the portfolio trading problem and derive an optimal trading strategy. Section 2.1 provides a brief description of previous studies that formulate the stochastic components of the financial market. Section 2.2 reviews previous studies that discuss heuristic methods for deriving an optimal trading strategy. Section 2.3 reviews previous studies that address the stochastic optimization problem to derive an optimal trading strategy using RL.

2.1 Stochastic programming-based models

Early studies on portfolio trading and, sometimes, management used stochastic programming-based models. Stochastic programming models formulate a sequence of investment decisions over time that can maximize a portfolio manager’s expected utility up to the end of the investment horizon. Golub et al. [15] modeled an interest rate series as a binomial lattice scenario using Monte Carlo procedures to solve a money management problem with stochastic programming. Kouwenberg [19] solved an asset-liability management problem using the event tree method to generate random stochastic programming coefficients. Consigli and Dempster [10] used scenario-based stochastic dynamic programming to solve an asset-liability management problem. However, stochastic programming-based models have the limitation of needing to generate numerous scenarios to solve a complex problem, such as understanding a financial environment, resulting in a large computational burden.

2.2 Heuristic methods

Because of this limitation of stochastic programming-based models, many studies have devised heuristic methods (i.e., trading heuristics). One of the most famous such methods is technical analysis for asset trading. This method provides a simple and sophisticated way to identify hidden relationships between market features and asset returns through the study of historical data. Using these identified relationships, investments are made in assets by taking appropriate positions. Brock et al. [5] conducted backtests with real and artificial data using moving average and trading range strategies. Zhu and Zhou [39] considered theoretical rationales for using technical analysis and suggested a practical moving average strategy to determine a portion of investments. Chourmouziadis and Chatzoglou [9] suggested an intelligent stock-trading fuzzy system based on rarely used technical indicators for short-term portfolio trading. Another popular heuristic method is the pattern matching (i.e., charting heuristics) method, which detects critical market situations by comparing the current series of market features to meaningful patterns in the past. Leigh et al. [21] developed a trading strategy using two types of bull flag pattern matching. Chen and Chen [8] proposed an intelligent pattern-matching model based on two novel methods in the pattern identification process. The other well-known heuristic method is a metaheuristic algorithm that can find a near optimal solution in acceptable computation time. Derigs and Nickel [13] developed a decision support system generator for portfolio management using simulated annealing, and Potvin et al. [32] applied genetic programming to generate trading rules automatically. Chen and Yu [7] used a genetic algorithm to group stocks with similar price series to support investors in making more efficient investment decisions. However, these heuristic methods have limited ability to fully search a very large feasible solution space because they are inflexible. Thus, we need to be careful about the reliability of obtaining an optimal trading strategy using these methods.

2.3 Reinforcement learning-based methods

A recent research direction is optimizing a trading strategy using RL such that a learning agent develops a policy while interacting with the financial environment. Using RL, a learning-based method, the learning agent can search for an optimal trading strategy flexibly in a high-dimensional environment. Unlike supervised learning, RL allows learning from experience, leading to training the agent with unlabeled data obtained from interactions with the environment.

In the earliest such studies, Neuneier [26, 27] optimized multi-asset portfolio trading using Q-learning, a model-free and value-based RL. In other early studies, Moody et al. [25] and Moody and Saffell [24] used Direct RL with Recurrent RL as a base algorithm and derived a multi-asset long-short portfolio trading strategy and a single-asset trading rule, respectively. Direct RL is policy-based RL, which optimizes an objective function by adjusting policy parameters, and Recurrent RL is an RL algorithm in which the last action is received as an input. These studies introduced several measures, such as the Sharpe ratio and the Sterling ratio, as objective functions and compared the trading strategies derived using different objectives. Casqueiro and Rodrigues [6] derived a single-asset trading strategy using Q-learning, which can maximize the Sharpe ratio. Dempster and Leemans [11] developed an automated foreign exchange trading system using an adaptive learning system with a base algorithm of Recurrent RL by dynamically adjusting a hyper-parameter depending on the market situation. O et al. [28] proposed a Q-learning-based local trading system that categorized an asset price series into four patterns and applied different trading rules. Bertoluzzo and Corazza [3] suggested a single-asset trading system using Q-learning with linear and kernel function approximations. Eilers et al. [14] developed a trading rule for an asset with a seasonal price trend using Q-learning. Zhang et al. [38] derived a trading rule generator using extended classifier systems combined with RL and a genetic algorithm. Almahdi and Yang [1] suggested a Recurrent RL-based trading decision system that enabled multi-asset portfolio trading and compared the performance of the system when several different objective functions were adopted. Pendharkar and Cusatis [30] suggested an indices trading rule derived using two different RL methods, on-policy (SARSA) and off-policy (Q-learning) methods and compared the performance of these two methods, and it also compared the performances of discrete and continuous agent action space modeling. Almahdi and Yang [2] used a hybrid method that combined Recurrent RL and particle swarm optimization to derive a portfolio trading strategy that considers real-world constraints.

More recently, DRL, which combines deep learning and RL algorithms, was developed, and, thus, studies have suggested using DRL-based methods to derive portfolio trading strategies. DRL methods enable an agent to understand a complex financial environment through deep learning and to learn a trading strategy by automatically applying an RL algorithm. Jiang et al. [18] used a deep deterministic policy gradient (DDPG), an advanced method of combining policy-based and value-based RL, and introduced various DNN structures and techniques to trade a portfolio consisting of cash and several cryptocurrencies. Deng et al. [12] derived an asset trading strategy using a Recurrent RL-based algorithm and introduced a fuzzy deep recurrent neural network that used fuzzy representation to reduce uncertainty in noisy asset prices and used a deep recurrent neural network to consider the previous action and utilize high-dimensional nonlinear features. Jeong and Kim [17] derived an asset trading rule that determined actions for assets and the number of shares for the actions taken. To learn this trading rule, Jeong and Kim [17] used a deep Q-network (DQN) with a novel DNN structure consisting of two branches, one of which learned action values while the other learned the number of shares to take to maximize the objective function.

The above studies used various RL-based methods in different problem settings. All of the methods performed well in each setting, but some issues limit the applicability of these methods to the real world. First, some problem settings did not consider transaction costs [3, 14, 17, 28, 30]. A trading strategy developed without assuming transaction costs is likely to be impractical for application to the real world. The second issue is that some strategies consider trading for only one asset [1, 3, 6, 11, 14, 12, 17, 24, 38]. A trading strategy of investing in only one risky asset may have high risk exposure because it has no risk diversification effect. Finally, in previous studies deriving multi-asset portfolio trading strategies using RL, the agent’s action space was defined as the portfolio weights in the next period [1, 2, 18, 25]. The action spaces of these studies do not provide portfolio traders with a direct guide that is applicable to a real-world trading scenario that includes transaction costs. This is because there are many different ways to transition from the current portfolio weight to the next portfolio weight. Thus, previous studies using portfolio weights as the action space required finding a way to minimize transaction costs at each rebalancing moment. Rebalancing in a way that reduces both transaction costs and dispersion from the next target portfolio is not an easily solved problem [16]. In addition, a portfolio trading strategy derived based on the action spaces of the previous studies may be difficult to apply to real-world trading because the turnover rate is likely to be high. An action space that determines portfolio weights can result in frequent asset switching because the amount of asset changes has no upper bound. Thus, we contribute to the literature by deriving a portfolio trading strategy that has no such issues.

3 Problem definition

In this study, we consider a portfolio consisting of cash and several risky assets. All assets in the portfolio are bought using cash, and the value gained from selling assets is held in cash. That is, the agent cannot buy an asset without holding cash and cannot sell an asset without holding the asset. This type of portfolio is called a long-only portfolio, which does not allow short selling. Our problem setting also has a multiplicative profit structure in that the portfolio value accumulates based on the profits and losses in previous periods. We consider proportional transaction costs that are charged according to a fixed proportion of the amount traded in transactions involving buying or selling. In addition, we allow the agent to partially buy or sell assets (e.g., the agent can buy or sell half of a share of an asset).

We set up some assumptions in our problem setting. First, transactions can only be carried out once a day, and all transactions in a day are made at the closing price in the market at the end of that day. Second, the liquidity of the market is high enough that each transaction can be carried out immediately for all assets. Third, the trading volume of the agent is very small compared to the size of the whole market, so the agent’s trades do not affect the state transition of the market environment.

To apply RL to solve our problem, we need a model of the financial environment that reflects the financial market mechanism. Using the notations summarized in Table 1, we formulate a Markov decision process (MDP) model that maximizes the portfolio return rate in each period by selecting sequential trading actions for the individual assets in the portfolio according to time-varying market features (Table 2).

3.1 State space

The state space of the agent is defined as the weight vector of the current portfolio before the agent selects an action and the tensor that contains the market features (technical indicators) for the assets in the portfolio. This type of state space is similar to that used in a previous study [18]. That is, the state in period $t$ can be represented as below (Equations (1)-(3)):

[TABLE]

where $w_{t}^{\prime}$ denotes the weight vector of the current portfolio and $X_{t}$ represents the technical indicator tensor for the assets in the portfolio. For this tensor, we use five technical indicators for the assets in the portfolio, as below (Equations (4)):

[TABLE]

Every set of five technical indicators can be expressed as a matrix (Equations (5)), where the rows represent each asset in the portfolio and the columns represent the series of recent technical indicators in the time window. Here, if we set a time window of size $n$ (considering $n$-lag autocorrelation) and a portfolio of $I$ assets, the technical indicator tensor is an $(I,n,5)$-dimensional tensor, as in Figure 1.

[TABLE]

3.2 Action space

We define the action space to overcome the limitations of the action spaces in previous studies. Agent actions determine which assets to hold and which assets to sell or buy at prespecified a constant trading size. For example, if a portfolio includes two assets and the trading size is 10,000 $USD$, then the agent can select the action of buying 10,000 $USD$ of asset1 and selling 10,000 $USD$ of asset2. The action space includes the trading directions of buying, selling, or holding each asset in the portfolio, so the action space contains $3^{I}$ different actions. These actions are expressed in a vector form that includes trading directions for each asset in a portfolio. In addition, each trading direction $(sell,hold,buy)$ is encoded as $(-1,0,1)$, respectively. For example, an action that involves selling asset1 and buying asset2 can be encoded into the vector $(-1,1)$.

Because the trading actions for individual assets are carried out in fixed trading size, this action space is modeled as a discrete type. Although this discrete action space may not be able to derive a trading strategy that outperforms trading strategies derived using a continuous action space [30], this action space can provide a direct trading guide that a portfolio trader can follow in the real world. Furthermore, this discrete action space can derive a portfolio trading strategy with lower turnover relative to the strategies developed in previous studies. In previous studies, if a portfolio with a very large amount of capital is changed by a small amount in portfolio weight then the trader may pay significant transaction costs. In addition, the losses from these transaction costs can be very high because portfolio weight changes have no upper bound. In contrast, our action space has an upper bound for portfolio weight changes, and, thus, the issue of massive changes in portfolio weights and the resulting large losses from transaction costs do not arise. Our agent action space has these advantages, and the only disadvantage of the fixed trading amount is similar to the restrictions of hedge funds that allow portfolio traders to trade below a certain amount each day. Thus, our discrete agent action space is not too unrealistic to apply to real-world trading.

3.3 MDP modeling

With the state space and action space defined in the previous subsections, we can define the MDP model as follows. The financial market environment operates according to this model during the investment horizon. To define the transitions in the financial market environment (i.e., the system dynamics in the MDP model), we need to define following parameters and equations:

[TABLE]

where $w_{t}$ denotes the portfolio weight after the agent takes an action at the end of period $t$ (Equation (6)). Equation (7) provides the constraint that the portfolio weight elements sum to one in all periods. Equations (8) and (9) represent the change in the portfolio value and the change in the proportions of the assets in the portfolio given the changes in the value of each asset in the portfolio, respectively. Here, $\odot$ represents the elementwise product of two vectors, and $\vec{1}$ is a vector of size I+1 with all elements equal to one. $\phi(\cdot)$ is an operator that not only increases a vector’s dimension by positioning zero as the first element but also adds it to the $\vec{1}$ vector ($\phi$ : $(e_{1},e_{2},...,e_{I})^{T}\rightarrow(1,{e_{1}}+1,{e_{2}}+1,...,{e_{I}}+1)^{T}$).

Now, we can define the state changes after the agent takes an action as follows:

[TABLE]

After the agent takes an action, transaction costs arise, and the portfolio value is then decayed (Equations (10)-(11)). Here, ${\mid}S{\mid}$ is the size of set $S$. $\hat{w}_{t,i}^{\prime}$ denotes the auxiliary weight of the portfolio that is needed to connect the change in the portfolio weights before and after the agent takes an action at the end of period $t$ (Equation (12)). The procedure by which the action selected by the agent is handled for trading in the financial environment is as follows. The auxiliary weight of an asset in the portfolio increases (or decreases) as a proportion of the trading size when buying (or selling) the asset. On the contrary, the auxiliary weights of the assets do not change when the agent holds the assets (Equation (13)). As a result of selling asset, the proportion of cash increases by the proportion of the trading size discounted by the selling transaction cost rate. As a result of buying asset, the proportion of cash decreases by the proportion of the trading size multiplied by the buying transaction cost rate (Equation (14)). To ensure that the sum of the portfolio weight elements equals one after the agent takes an action, a process for adjusting the auxiliary weights is required (Equation (15)). In summary, the financial market environment transition is illustrated by Figure 2.

Last, the reward in the MDP model should reflect the contribution of the agent’s action to the portfolio return. This reward can be simply defined as the portfolio return. However, if the portfolio return is defined only as a reward, then different reward criteria can be given depending on the market trend. For example, when the market trend is sufficiently improving, then no matter how poor the agent’s action is, a positive reward is provided to the agent. In contrast, if the market trend is sufficiently negative, then no matter how helpful the agent’s action is, a negative reward is provided to the agent. Thus, the reward must be defined as the rate of change in the portfolio value by which the market trend is removed. Therefore, we define the reward as the change in the portfolio value at the end of the next period relative to the static portfolio value (Equation (16)). The static portfolio value is the next portfolio value when the agent takes no action at the end of the current period (Equation (17)).

[TABLE]

4 Methodology

In this section, we introduce our proposed approach for deriving the portfolio trading strategy using DQL. In our action space, some issues may prohibit a DQL agent from deriving an intelligent trading strategy. We first explain how to resolve these issues by introducing some techniques and applying existing methodologies. Then, we describe our DQL algorithm with these techniques.

4.1 Mapping function

In our action space, some actions are infeasible in some states (e.g., the agent cannot buy assets because of a cash shortage or cannot sell assets because of a shortage of held assets). To handle infeasible actions, we first set the action values (i.e., Q-values) of infeasible actions to be very low to mask these actions [20]. Thus, we need to define a rule for selecting the appropriate action from the remaining actions when infeasible actions are excluded. For this rule, a simple way in which the agent selects the largest Q-valued one among remaining actions can be considered [37]. However, this simple rule can result in an unreasonable trading strategy. For example, when an agent’s strategy selects the action of selling both asset1 and asset2 but this action is infeasible owing to a lack of asset2, the action of buying both asset1 and asset2, which is the largest Q-value action in the remaining action space, is selected. Because learning the similarity between actions is difficult for an RL agent, the agent will take this action without any doubt even though this selected action is the opposite of the original action determined by the agent’s strategy. This issue leads to the selection of unreasonable actions, which degrades the trading performance. A mapping rule is required to map infeasible actions to similar and valuable actions in the feasible action set. Thus, we resolve this issue by introducing a mapping function that contains several mapping rules.

The mapping function is a type of constraint on action space in RL that allows the agent to derive a reasonable trading strategy. Pham et al. [31], Bhatia et al. [4] handled constrained action space by adding an optimization layer, so-called OptLayer, for solving quadratic programming at the last layer of the agent’s policy network, determining an action that minimizes differences from the output at the previous layer while satisfying constraints. However, the method cannot be applied directly to our situation because they can only deal with continuous action space. Although OptLayer can be applied for handling our constrained action space by revising the integer quadratic programming (IQP) version, the solution of IQP suffers multiple-choice issue in this situation because there are tied actions which have the smallest distance from the original action. To overcome the limitation of the OptLayer, we devise the mapping function based on a heuristic searching method. The function maps to one feasible action which has the largest Q-value among tied actions that have the smallest distance from the original action. Moreover, the computation costs of the mapping function are lower than those of OptLayer. Therefore, the mapping function is an extended efficient method from the line of study to handle infeasible action using the concept of distance between actions, such as OptLayer.

The mapping function contains two mapping rules, each of which is required for mapping infeasible actions, that are divided into two cases. In the first case, the amount of cash is not sufficient to take an action that involves buying assets. In this case, a similar action set is derived by holding rather than buying a subset of the asset group to be bought in the original action. Thereafter, infeasible actions are mapped to the most valuable feasible actions in the similar action set. For example, if the action of buying both asset1 and asset2 is infeasible owing to a cash shortage, this action is mapped to the most valuable feasible action within the set of similar actions, which includes the action of buying asset1 and holding asset2, the action of holding asset1 and buying asset2, and the action of holding both asset1 and asset2. In the second case, an action that involves selling assets is infeasible because of a shortage of the assets. In this case, the original action is simply mapped to an action in which the assets that are not enough to sell are held. These examples are illustrated in Figure 3.

We provide the details of the two mapping rules and the mapping function in the following pseudocode in Algorithm (1). In the Algorithm, the last part (i.e., Lines (26)-(27)) of the mapping rule for the second case(Rule2) is necessary. Because, in the second case, converting the original action of selling assets that cannot be sold into an action that holds the selling assets which cannot be sold, then the cash amount gained from selling assets is removed, causing the first infeasible action case to arise. Furthermore, this part of the code can handle the special case in which an asset shortage and a cash shortage occur simultaneously. Next, the RL flow chart with the mapping function technique is shown in Figure 4.

4.2 DQN algorithm

We optimize the multi-asset portfolio trading strategy by applying the DQN algorithm. DQN is the primary algorithm for DQL. Mnih et al. [22] developed the DQN algorithm, and Mnih et al. [23] later introduced additional techniques and completed this algorithm. The base algorithm for DQN, Q-learning, is value-based RL, which is a method that approximates an action value (i.e., a Q-value) in each state. Further, Q-learning is a model-free method such that even if the agent does not have knowledge of the environment, the agent can develop a policy using repeated experience by exploring. In addition, Q-learning is an off-policy algorithm, that is, the action policy for selecting the agent’s action is not the same as the update policy for selecting an action on the target value. An algorithm based on Q-learning that approximates the Q-function using DNN is the basis of DQN [22]. To prevent DNN from learning only through the experience of a specific situation, experience replay was introduced to sample a general experience batch from memory. Additionally, the DQN algorithm used two separate networks: a Q-network that approximates the Q-function and a target network that approximates the target value needed for the Q-network updated to follow a fixed target [23]. Based on this algorithm, we introduce several techniques to support the derivation of an intelligent trading strategy.

The existing DQN algorithm updates the Q-network with experience by allowing the agent to take only one action in each stage. Because the agent has no information about the environment, only one action is taken then proceeding to the next state. Thus, it is impossible to take multiple actions in the existing DQN. However, for this problem, we use historical technical indicator data of the assets in the portfolio as training data. Thus, our agent can take multiple actions in one state in each stage and observe all of their experiences based on those actions. To utilize this advantage, we introduce a technique that simulates all feasible actions in one state at each stage and updates the trading strategy by using the resulting experiences from conducting these simulations.

Motivated by Tan et al. [35], we utilize a simulation technique that takes all feasible actions virtually to force to the agent learns about many experiences efficiently for deriving a fully searched multi-asset trading strategy. Thus, this technique can relax the data shortage issue that arises when deriving a multi-asset trading strategy. Although simulating all feasible actions can result in a huge computational burden, using multi-core parallel computing can prevent this computational burden from greatly increasing. Moreover, even if the agent takes multiple actions in the current state, the next state only depends on the action selected by the action policy (epsilon-greedy) with the mapping rule. The application of this technique requires a change in the data structure of the element in replay memory for storing a list of experiences in a state. The concepts related to this technique are illustrated in Figure 5. In this figure, $a_{t}^{j}$ means that the $j-th$ action of the agent is taken at the end of period $t$. $r_{t}^{j}$ is the reward obtained by taking action $a_{t}^{j}$, and $s_{t+1}^{j}$ is the next state that results from taking action $a_{t}^{j}$.

In DQN, a multiple output neural network is commonly adopted as the Q-network structure. In this network structure, the input of the neural network is the state, and the output is the Q-value of each action. Using the above technique, we can approximate the Q-value of all feasible actions by updating this multiple output Q-network in parallel with the experience list. To maintain Q-values of infeasible actions, the current Q-value of an infeasible state-action pair is assigned to the target value of the Q-network output of the corresponding infeasible action to set a temporal difference error of zero. Furthermore, as in DQN, several experience lists are sampled from replay memory, and the Q-network is updated using the experience list batch. A detailed description of the process for updating the Q-network is shown in Figure 6.

In addition, to apply RL, learning episodes must be defined for the agent to explore and experience the environment. Rather than defining all of the training data, which cover several years, as one episode, we divide the training data into several episodes. If we define a much longer training episode than the investment horizon of the test data that will be used to test the trading strategy, this difference in the lengths of the training and test data can produce negative results. For example, in our experiment, the training and testing processes begin with the same portfolio weights. In this case, the farther the agent is from the beginning of the long training episode, the farther the agent is from the initial portfolio weights. Thus, it is difficult for the agent to utilize the critical experience obtained from the latter half of the long episode in the early testing process. Therefore, we divide training data into sets of the same length as the investment horizon of the test data (i.e., one year, as the investment horizon of the test data is a year in our experiment). Thus, the criteria for dividing the training data are defined in yearly units so that the episodes do not overlap (e.g., episode1 contains data from 2016, and episode2 contains data from 2015). In each training epoch, the agent explores and learns in an episode sampled from the training data.

It is well known that more recent historical data have more explainable for predicting future data than less recent historical data have. Thus, it is reasonable to assign higher sampling probabilities to episodes that are closer to the test data period [18]. We use a truncated geometric distribution to assign higher sampling probabilities to episodes that are closer to the test period. This truncated geometric sampling distribution is expressed in Equation (18). Here, $y$ is the year of the episode, $y_{v}$ is the year of the test data, and $N$ is the number of total training episodes. $\beta$ is a parameter for this sampling distribution that ranges from zero to one. If this parameter is closer to one, episodes closer to the test period are sampled frequently.

[TABLE]

To implement DQN, we need to model the neural network structure for approximating the Q-function of an agent’s state and action. We construct a hybrid encoder LSTM-DNN neural network that enables us to approximate the Q-value of an agent’s action in our predefined state and action space. First, we train LSTM autoencoder, an unsupervised learning method for compressing sequence data, for identifying latent variables of the historical sequence of predefined technical indicators of assets in the portfolio [34]. Then the autoencoder is fitted in the historical pattern of technical indicators of assets, the decoder is removed and the encoder keeps as a standalone model that encodes the technical indicator sequences for assets to lower dimension latent variables. Each asset in the portfolio shares the same encoder LSTM to take the sequence encoding procedure because it is known that a single deep learning model is more effective for learning the feature patterns of different assets than multiple deep learning models that learning individual assets [33]. Then, the encoded outputs for each asset are concatenated to create the intermediate output, and this intermediate output is then combined again with the current portfolio weights to use as the input to the DNN. Through this DNN layers, we can obtain the Q-value of each action of the agent. Because these DNN layers extract meaningful features through nonlinear mapping using a multi-layer neural network and conducts a regression for the Q-value, we refer to these layers as the DNN regressor. The overall Q-network structure is as shown in Figure 7. In summary, the overall DQN algorithm for our approach for deriving the portfolio trading strategy is as follows Algorithm (2). In offline learning, we use this algorithm to build an initial trading strategy fitted on historical data and in online learning, adapt the trading strategy by updating based on the daily observed data in the trading process.

5 Experimental results

In this section, we demonstrate that the DQN strategy (i.e., the trading strategy derived using our proposed DQN algorithm for portfolio trading) can outperform in real-world trading. We conduct a trading simulation for two different portfolio cases using both our DQN strategy and traditional trading strategies as benchmarks, and we verify that the DQN strategy is relatively superior to the other benchmark strategies based on several common performance measures.

5.1 Performance measures

We use three different output performance measures to evaluate trading strategies. The first measure is the cumulative return based on the increase in the portfolio value at the end of the investment horizon relative to the initial portfolio value, as defined as Equation (19):

[TABLE]

where $t_{f}$ is the final date of the investment horizon and $P_{0}$ is the initial portfolio value.

The second measure is the Sharpe ratio, as defined in Equation (20):

[TABLE]

where $std({\rho}_{t})$ is the standard deviation of the daily return rate, $\rho_{f}$ is the daily risk-free rate (assumed to be 0.01%), and annualization term (square root of the number of annual trading days) is multiplied. This ratio is a common measure of the risk-adjusted return, and it is used to evaluate not only how high the risk premium is but also how small the variation in the return rate is.

For the last measure, we use the customized average turnover rate defined as in Equation (21):

[TABLE]

The average turnover measures the average rate of change of the portfolio weight vector during the investment horizon. We do not have to consider changes in the cash proportion, so we customize this measure by excluding the change in the weight on cash before and after the agent takes an action. This rate can evaluate the change in the proportions of asset investments. Considering transaction costs, this measure should be low to better apply the trading strategy in the real world.

5.2 Data summary

We experiment with two different three-asset portfolios. The first consists of three exchange traded funds (ETFs) in the US market that track the S&P500 index, the Russell 1000 index, and the Russell Microcap Index. This type of portfolio was tested in a previous study [1]. The second portfolio is a Korean portfolio consisting of the KOSPI 100 index, the KOSPI midcap index, and the KOSPI microcap index. More information for these test portfolios is provided in Table 3.

We obtain data on the three US ETFs from Yahoo Finance and data on the Korean indices from Investing.com. Both cases are tested in 2017. The trading strategy for the US portfolio is derived by training on data from 2010 to 2016, and the trading strategy for the Korean portfolio is derived by training on data from 2012 to 2016.

5.3 Experiment setting

Through several rounds of tuning, we derive appropriate hyper-parameters. In particular, the time window size$(n)$ is the most important hyper-parameter, and we adopt the value of 20 among the candidates (5,20,60,120). This time window size and the other tuned hyper-parameters are summarized in Table 4.

In the experiment, we also need to set trading parameters, such as the initial portfolio value and the trading size. We set the initial portfolio value as one million in both portfolio cases (e.g., 1M $USD$ for the US portfolio and 1M $KRW$ for the Korean portfolio). Similarly, we set the trading size as ten thousand in both portfolio cases (e.g., 10K $USD$ trading size for the US portfolio case and 10K $KRW$ for the Korean portfolio case). We set the transaction cost rate for buying and selling in both the US and Korean markets as 0.25%. In both cases, the initial portfolio is set up as an equally weighted portfolio, in which every asset and cash has the same proportion.

5.4 Benchmark strategy

To evaluate our DQN strategy, we compare it to some traditional portfolio trading strategies. The first strategy is a buy-and-hold strategy $(B\&H)$ that does not take any action but rather holds the initial portfolio until the end of the investment horizon. The second strategy is a randomly selected strategy $(RN)$ that takes action within the feasible action space randomly in each state. The third strategy is a momentum strategy $(MO)$. This strategy buys assets whose values increased in the previous period and sells assets whose values decreased in the previous period. However, if it cannot buy all assets with increased values, it gives buying priority to assets whose values increased more. If it is unable to sell assets whose values decreased, it simply holds the assets. The last strategy is a reversion strategy $(RV)$, which is the opposite of the momentum strategy. This strategy sells assets whose values increased in the previous period and buys assets whose values decreased in the previous period. However, if it cannot buy all of the assets whose values decreased, it gives buying priority to the assets whose values decreased more. If it is unable to sell the assets whose values increased, then it simply holds the assets.

5.5 Result

We derive a trading strategy for both portfolio cases using DQN. For both cases, we identify the increase in the cumulative return over the investment horizon of the test period as episode learning continues. Figure 8 shows the trend in the cumulative return performance over the learning episodes in both cases.

Table 5 shows the number of changes in the phase of trading direction (i.e., the number of changes from selling to buying or vice versa, except for holding) before and after applying the mapping function. A trading strategy that has frequent changes in the phase of trading action implies an unreasonable trading strategy because these changes deteriorate trading performance by incurring meaningless transaction costs. Through the experimental result, we identify that when the mapping function is applied to the DQN strategy, the number of changes in the phase of the trading direction decreases relative to when it is not applied. Following by decreasing the number of changes in the phase of the trading direction, the cumulative return of the trading strategy increases by 6.68% relative to the one without applied the mapping function in the US portfolio case. Likewise, in the Korean portfolio case, the cumulative return of the trading strategy increases by 10.83% relative to the one without applied the mapping function. Therefore, we demonstrate that the mapping function contributes to deriving a reasonable trading strategy.

Figure 9 shows the portfolio value trend when applying the DQN strategy and the benchmark strategies in the US and Korean portfolio cases. In the US portfolio case, we observe that the DQN strategy outperforms the benchmark strategies for most of the test period. The final portfolio value of the DQN strategy is 15.69% higher than that of the B&H strategy, 33.74% higher than that of the RN strategy, 21.81% higher than that of the MO strategy, and 114.47% higher than that of the RV strategy. Likewise, in the Korean portfolio case, we observe that the DQN strategy outperforms the benchmark strategies for most of the test period. The final portfolio value of the DQN strategy is 25.52% higher than that of the B&H strategy, 34.99% higher than that of the RN strategy, 13.22% higher than that of the MO strategy, and 247.91% higher than that of the RV strategy.

Table 6 summarizes the output performance measure results when using DQN and the benchmark strategies in both portfolio cases. This table shows that the DQN strategy has the best cumulative return and Sharpe ratio performances for the US portfolio, and this strategy has the lowest turnover rate except for the B&H strategy, which has no turnover rate. In the Korean portfolio case, the DQN strategy also has the best cumulative return and Sharpe ratio performances. Moreover, the DQN strategy has the lowest turnover rate except for the B&H strategy. Given that the B&H strategy does not incur any transaction costs during the investment horizon, it is a remarkable achievement that the DQN strategy outperforms the B&H strategy in terms of the cumulative return and Sharpe ratio.

6 Conclusion

The main contribution of our study is applying the DQN algorithm to derive a portfolio trading strategy on the practical action space. However, applying DQN to portfolio trading has some challenges. To overcome these challenges, we devise a DQL model for trading and several techniques. First, we introduce a mapping function for handling infeasible actions to derive a reasonable trading strategy. Trading strategies derived from RL agents can be unreasonable to apply in the real world. Thus, we apply a domain knowledge rule to develop a trading strategy with an infeasible action mapping constraint. As a result, this function works well, and we can derive a reasonable trading strategy. Second, we design DQL agent and Q-network for considering multi-asset features and derive a multi-asset trading strategy in the practical action space, determining the trading direction of the asset, by overcoming the dimensionality problem. Third, we relax the data shortage issue for deriving well-fitted multi-asset trading strategies by introducing a technique that simulates all feasible actions and then updating the trading strategy based on the experiences of these simulated actions.

The experimental results show that our proposed DQN strategy is a superior trading strategy relative to benchmark strategies. Based on the results of the cumulative return and the Sharpe ratio, the DQN strategy is more profitable with lower risk than other benchmark strategies. In addition, based on the results of the average turnover rate, the DQN strategy is more suitable for application in real-world trading than benchmark strategies.

However, our proposed methodology still has a limit of scalability, arising the dimensionality problem if the number of assets in the portfolio very large. Furthermore, in our study, the reward of the MDP model is optimized only for returns and not for risk. Nevertheless, the contributions of this study are still valuable because of the novel techniques for expressing the practical applicability of the portfolio trading strategy. By responding to the limit of the current study, we will try to devise the method for resolving these limitations in future research.