Deep Reinforcement Learning for Optimal Asset Allocation Using DDPG with TiDE

TL;DR

This paper introduces a novel reinforcement learning approach using DDPG with TiDE for dynamic asset allocation, outperforming traditional methods and passive strategies in financial markets.

Contribution

It integrates TiDE into DDPG to enhance continuous decision-making in asset allocation, addressing limitations of conventional methods and demonstrating superior empirical performance.

Findings

DDPG-TiDE outperforms Q-learning in asset allocation tasks.

DDPG-TiDE achieves higher risk-adjusted returns than buy-and-hold.

The framework effectively models the sequential decision-making process in finance.

Abstract

The optimal asset allocation between risky and risk-free assets is a persistent challenge due to the inherent volatility in financial markets. Conventional methods rely on strict distributional assumptions or non-additive reward ratios, which limit their robustness and applicability to investment goals. To overcome these constraints, this study formulates the optimal two-asset allocation problem as a sequential decision-making task within a Markov Decision Process (MDP). This framework enables the application of reinforcement learning (RL) mechanisms to develop dynamic policies based on simulated financial scenarios, regardless of prerequisites. We use the Kelly criterion to balance immediate reward signals against long-term investment objectives, and we take the novel step of integrating the Time-series Dense Encoder (TiDE) into the Deep Deterministic Policy Gradient (DDPG) RL framework for continuous decision-making. We compare DDPG-TiDE with a simple discrete-action Q-learning RL framework and a passive buy-and-hold investment strategy. Empirical results show that DDPG-TiDE outperforms Q-learning and generates higher risk adjusted returns than buy-and-hold. These findings suggest that tackling the optimal asset allocation problem by integrating TiDE within a DDPG reinforcement learning framework is a fruitful avenue for further exploration.

Tables

Table 1. Table 1: Macroeconomic and financial predictors for market environment indicators.

Predictor	Description
dp	dividend-price ratio
ep	earnings-price ratio
bm	book-to-market ratio
ntis	net equity expansion
tbl	Treasury-bill rate
tms	term spread
dfy	default spread
infl	inflation
corpr	high-quality corporate bond rate
ltr	long-term rate of return
svar	stock variance

Table 2. Table 2: Performance metrics.

Metric	Calculation
Logarithmic Utility (cumulative rewards)	$\text{LU}=\ln \left(W_{t+1}\right)={\sum }_{t=1}^T\ln \left(1+r_t\right)={\sum }_{t=1}^{T}rewar{d}_t$
Portfolio Value (cumulative returns)	$\text{PV}=\frac{W_{t+1}}{W_0}$
Sharpe Ratio (12-month rolling window)	$\text{SR}=\frac{{𝔼}_t\left[{\rho }_t-{\rho }_f\right]}{\sqrt{{\mathrm{var}}_t\left[{\rho }_t-{\rho }_f\right]}}$

Equations

$$U(W)=\frac{W^{1-R_a}}{1-R_a}$$

$$\ln U(W)=(1-R_a)\ln (W)-\ln (1-R_a)$$

$$\max E[\ln U(W)]=\max E[(1-R_a)\ln (W)-\ln (1-R_a)]=\max E[\ln (W)]$$

$$Rewar{d}_t=\gamma \ln (1+r_t)$$

$$\sum _{t=1}^{T}Rewar{d}_t=\sum _{t=1}^T\ln (1+r_t{)}^{\gamma }=\ln \prod _{t=1}^T(1+r_t{)}^{\gamma }$$

$$W_0\prod _{t=1}^T(1+r_t{)}^{\gamma }=W_T,\text{ where assuming }{W}_0=1.$$

$$\max E[\ln W_0+\sum _{t=1}^{T}Rewar{d}_t]=\max E[\ln W_0\prod _{t=1}^T(1+r_t{)}^{\gamma }]=\max E[\ln W_t]$$

$$Q^{\ast }(s,a)=\mathbb{E}\left[r+\gamma \underset{a^{\prime }}{\max }{Q}^{\ast }(s^{\prime },a^{\prime })\mid s,a\right]$$

$$Q(s,a)\leftarrow Q(s,a)+\alpha \left(r+\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })-Q(s,a)\right)$$

$${\nabla }_{\theta }J=\mathbb{E}\left[{\nabla }_a{Q}_{\varphi }(s,a){\nabla }_{\theta }{\pi }_{\theta }(s)\right]$$

Full text

\email

rongwei.liu@bristol.ac.uk

1 Introduction

Financial markets comprise a diverse array of assets competing for investors’ limited capital. Asset returns, characterized by volatility, do not necessarily equate to a market risk premium but rather reward exposure to systematic risk. Diversified portfolios aid in managing idiosyncratic risk, with the market portfolio embodying non-diversifiable uncertainty that influences asset prices. While the true market portfolio remains unobservable, it is commonly approximated by a broad stock market index. This study centers on optimizing asset allocation between a risk-free asset and the market portfolio. Rather than estimating the efficient frontier or determining the exact tangency point of the Capital Market Line, the focus is on identifying the investor-specific optimal allocation to maximize long-term portfolio performance, assuming the market portfolio is observable and broadly representative of the investable universe.

The classical optimal portfolio methods are mean variance optimization (MVO) [17], which was proposed to construct portfolios. Several tools have been used to estimate the distribution of returns on assets and to select valuable portfolios. Because the true distribution of returns is unknown, the population is estimated and substituted by the sample, and then the optimal portfolio weights are selected based on the conclusions derived. The solution of portfolio weights that maximize utility depends on the first-order and second-order returns moments and the investor’s risk aversion coefficient [12]. This method is known as plug-in estimation. The limitation of this approach is that it assumes a consistent estimate of population statistics, but the estimation results will not converge when the assets in the portfolio gradually increase.

The hyperparameters of machine learning (ML) models can be tuned to maximize the expected out-of-sample utility or minimize a predefined risk function, provided that the first-order and second-order moments and the functional form of the return distribution are known [23]. First, a class of equations with hyperparameters is selected, the samples are trained, and the hyperparameters are estimated. Second, the hyperparameters are tuned with out-of-sample data on the validation set to get the best performance. This is a two-step approach. For portfolio selection methods, cutting-edge machine learning directly considers the integration of utility maximization into statistical problems for weighted function estimation [28]. This method is known as the one-step procedure or parametric portfolio weight approach. It is not necessary to know the distribution assumptions of the returns and, instead, only the investment utility function and the weight function are needed. An intuitive example is the Maximum Sharpe Ratio Regression (MSRR) [3], which is adaptable to sophisticated machine learning models such as neural networks [24].

Reinforcement learning (RL) uses a reward signal from a simulated market environment to iteratively guide learning, and has been widely used in many problems, including asset allocations [9]. In this paper, we introduce the utility equation based on the Kelly criterion and determine the reward through formula derivation to establish the optimal policy problem within the MDP framework. Additionally, we take the novel approach of incorporating the Time-series Dense Encoder (TiDE) [6] into the Deep Deterministic Policy Gradient (DDPG) [14] mechanism to enhance the handling of temporal relationships in multivariate time series within the state space.

The remainder of this paper is organized as follows. Section 2 reviews the classification of RL, the research gaps in actor-critic approaches, and the theoretical basis and methodological innovations in the solutions proposed in this study. Section 3 elucidates the optimal function according to the Kelly criterion within the MDP framework and explains the iterative principle behind DDPG with TiDE. The source data, experimental configurations, and performance metrics are also described. Section 4 presents the experimental results. Finally, Section 5 concludes and suggests potential avenues for future work.

2 State of the art

RL encompasses various algorithms distinguished by interaction paradigms (online vs. offline), policy update mechanisms (on-policy vs. off-policy), and model reliance (model-based vs. model-free). Although this classification offers a useful conceptual framework, its practical implications are particularly significant in financial contexts. Online RL, though theoretically appealing for its adaptability, can encounter latency and instability due to the high costs and risks associated with live market interactions. Conversely, offline RL allows learning from historical or simulated data, but struggles with generalization if the training data are not adequately representative. On-policy methods learn from the policy they aim to improve, ensuring internal consistency but limiting exploration. In contrast, off-policy methods like Q-learning [25] use data from different behavior policies, enhancing data efficiency, a crucial advantage in finance where real-time sampling is limited [20] [21]. Therefore, this study adapts the ideas underlying the success of Q-learning to the off-policy domain as the starting point for simulation. Model-based RL approaches, compared to model-free mechanisms, remain largely confined to simulating and modeling price–volume data and continue to suffer from the inherent suboptimality of the two-step process of prediction followed by optimization. Due to variations in data structures and forecasting models, the robustness and generalizability of such methods remain subject to further scrutiny [26] [27]. The datasets used in this paper cover more than twelve indicators. Consequently, this study focuses on model-free RL, which prevails in financial contexts. Within this domain, algorithms are generally divided into actor-only and critic-only approaches according to how the architectures represent decision-making. Actor-only methods (e.g., policy gradient) directly optimize policies but suffer from high variance and slow convergence, especially in noisy settings. In contrast, critic-only methods (e.g., Q-learning) offer greater stability through value estimation, but struggle with flexibility in continuous action spaces. Despite extensive study, the comparative advantages of these paradigms remain unclear due to inconsistent benchmarks and experimental setups in the literature [2] [29].

Actor-Critic (AC) architectures integrate policy learning (actor) with value estimation (critic) to improve sample efficiency and facilitate continuous decision-making [14]. A notable implementation within this framework is the DDPG mechanism, which employs neural networks for both actor and critic roles. The application of DDPG to portfolio optimization has been widely studied. Jiang et al. [11] introduced an RL framework incorporating CNN, RNN, and LSTM networks, all of which outperformed traditional benchmarks in cryptocurrency markets. Liang et al. [13] compared DDPG, PPO, and PG on five randomly selected stocks from the Chinese market, concluding that PG was preferable. Pham et al. [22] investigated multi-agent RL using 10 popular stocks from the Ho Chi Minh Stock Exchange and VN30 Index Futures, demonstrating superior performance over the risk-free rate and the potential for cross-hedging strategies. Cong et al. [5] developed AlphaPortfolio, a multi-sequence, attention-based model for direct investment target optimization, finding that the Sharpe ratio is time delayed compared to other reward settings. Aboussalah et al. [1] applied CNN-based RL to asset allocation, noting limited success with stable algorithms and proposing a more interpretable strategy by integrating the Kelly criterion. Although DDPG has been widely applied in asset allocation tasks, many studies are limited to specific markets or rely on a small subset of randomly chosen stocks. This narrow focus undermines the generalizability of their conclusions and limits their relevance to broader investor populations. Moreover, a critical shortcoming in the existing application of RL to portfolio optimization lies in the insufficient attention paid to reward design within the MDP framework. Many prior works implicitly assumed that maximizing cumulative Sharpe ratios is equivalent to identifying an optimal strategy but have failed to account for the non-additive and time-lagged nature of this metric, potentially leading to mismatches between the learning signal and true investor objectives.

This study seeks to fill the above research gaps by examining portfolio allocation between a market index—the CRSP index, serving as a representative proxy for the aggregate U.S. equity market—and a risk-free asset. To ensure theoretical rigor, the study adopts the Kelly criterion through the use of the constant relative risk aversion (CRRA) utility function and the accumulated reward, providing mathematical consistency between log-utility maximization and long-term capital growth. Furthermore, to better capture temporal dependencies among heterogeneous financial indicators, we take the novel step of integrating the TiDE model into the DDPG framework.111On 25/06/2025, R. Liu conducted a review of all 472 papers listed on GoogleScholar as papers that cited the original TiDE article [6]. A total of 440 articles were accessible to read: of these, 240 articles implemented TiDE in some capacity, while the others only mentioned TiDE in discussion. Of the studies that implemented TiDE, none incorporated TiDE within an RL framework and only 12 papers mentioned RL. Therefore, we conclude that no previous publication has used TiDE within the DDPG framework for RL. This architectural enhancement enables the agent to process richer financial information and learn more adaptive and robust portfolio policies. By conducting comparative experiments using DDPG with TiDE, Q-learning, and a buy-and-hold benchmark, we will assess the capacity of RL to generate investor-specific optimal allocations under long-horizon utility-based criteria. This establishes a stronger foundation for the development of data-driven, adaptive portfolio strategies in realistic financial environments.

3 Methods and Materials

3.1 Kelly Strategy and MDP Simulation

In 1971, Merton [19] explored the trading of stocks and bonds within his portfolio model to optimize the utility of investors. In 2010, MacLean [15] equated the investor’s optimal expected utility with the market’s maximum expected return, labeling the portfolio’s investment objective as the Kelly criterion, formulated as $\max E[\ln U(W)]=\max E[\ln Return]$, where $Return=1+r_{t}$ and $r_{t}$ is the rate of risky asset return $\sim\mathcal{N}\left(\mu,\sigma^{2}\right)$. Then, based on the derivation using Itô’s Lemma, the optimal investment fraction in the risky asset is calculated as $\pi^{*}=(\mu-r_{f})/R_{a}\sigma^{2}$, where $\mu$ is the expected return of the risky asset, $r_{f}$ is the return rate of the risk-free asset, $\sigma$ is the standard deviation, $R_{a}$ is the coefficient of constant risk aversion. However, the distribution of risky assets in real financial markets is complex and unknown, and its indicators are not limited to price data. Therefore, the optimal solution is limited by various conditions and cannot adapt to the dynamically adjusting market environment. Using MDP, RL algorithms are developed for the practical implementation of the RL Kelly strategy [10].

An MDP provides a mathematical framework for sequential decision making under uncertainty, defined by the tuple (S, A, P, R, $\gamma$), where S is the state space, A is the action space, P(s’ $\mid$ s, a) denotes transition probabilities, R(s, a) is the reward function, and $\gamma$ is the discount factor. In RL, an agent interacts with an MDP environment, learning from rewards and iteratively improving its policy. Within financial markets, the environment can be treated as an MDP whose states capture market conditions (e.g., historical macro factors, excess returns, and volatilities), whose actions represent trading decisions (e.g., buy, sell, hold, or adjusting risk-asset weights), and whose rewards measure long-term value (e.g., log-return). By simulating a variety of historical or synthetic market scenarios, an RL agent explores possible strategies, continuously refines its portfolio allocations through reward feedback, and strives to maximize cumulative logarithmic returns while respecting risk constraints. Through repeated trials, RL avoids strict rule-based heuristics and adapts dynamically to changing market conditions.

3.2 CRRA Utility and Reward Setting

In RL for portfolio optimization, the reward must provide immediate feedback on the chosen action but also embody the longer-term risk-return trade-off. One concise way to encode both dimensions is through a coefficient of constant relative risk-aversion (CRRA)222CRRA is a widely used assumption in economic models; and is assumed in Merton’s model [18]. utility function:

[TABLE]

where $W$ is wealth and $R_{a}$ is relative risk aversion (constant). When this expression reduces to logarithmic utility, then:

[TABLE]

Maximizing the expectations of both sides, since $0<R_{a}<1$, yields the following:

[TABLE]

Hence, setting,

[TABLE]

gives total reward,

[TABLE]

and therefore wealth at time T is,

[TABLE]

Hence, to maximize expected wealth, we have:

[TABLE]

which corresponds exactly to a log-utility perspective $\max E[\ln U(W)]$. Optimizing the expected sum (or discounted sum) of rewards aligns with maximizing the expected future log wealth, known as the Kelly criterion [16]. This formulation ensures that each action’s return-risk balance (via the immediate log-return feedback) contributes to the long-run objective of sustainable capital growth, thereby unifying immediate step-by-step reward signals with a principled, long-horizon utility measure.

3.3 Q-Learning and Bellman Equation

Since the true transition dynamics of time series is typically unknown, Q-learning [25] uses temporal difference (TD) learning to iteratively approximate $Q^{*}(s,a)$ using sampled experience $(s,a,r,s^{\prime})$. The update rule follows from the Bellman equation:

[TABLE]

The Bellman equation is the foundation of Q-learning, providing a recursive relationship to estimate the optimal action-value function $Q^{*}(s,a)$. The Bellman equation expresses the value of a state-action pair as the immediate reward plus the expected discounted return of the optimal future actions:

[TABLE]

where $\alpha$ is the learning rate that controls the magnitude of the update; $\gamma$ is the discount factor that weighs future rewards; and $r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})$ is the TD target, representing the optimal future return estimate. This update process ensures that Q-learning converges to $Q^{*}(s,a)$ with adequate exploration and appropriate scheduling of learning rates. In practice, Q-learning is particularly useful for discrete action spaces, such as financial trading decisions (buy/sell/hold), where it helps optimize trading strategies by learning from simulated market interactions.

In this work, the Q-learning algorithm is applied with a discretized observation and action space to balance the tractability of the model with sufficient flexibility. Specifically, the state space is obtained by performing K-means clustering ($k=50$) on key market features, yielding a finite set of cluster labels as the observed states. For actions, rather than using a simplistic buy-sell-hold triad, the agent discretizes portfolio weights $\omega\in\{0.0,0.1,\ldots,1.0\}$. This design allows for somewhat finer control over investment levels, while remaining computationally manageable. As a benchmark, this study includes the buy-and-hold strategy—invest all funds in the market index at the beginning and hold until the end—as a passive baseline with fixed $\omega=1.0$. By comparing against a passive strategy, we can ensure that any gains in performance achieved by Q-learning are due to actively rebalancing portfolio weights.

3.4 DDPG with TiDE

Q-learning is a value-based RL algorithm designed for discrete state-action spaces. In the described experiments, the market is represented by a K-means-based discretization into 50 clusters and logarithmic utility (Kelly criterion) is optimized. However, this approach suffers from two major challenges: the Q-table becomes impractically large as the number of clusters increases, and action selection by maximizing Q-values entails a costly search over all discrete bins. To address these limitations, policy-based methods were introduced, allowing RL agents to learn a policy $\pi(a|s)$ directly rather than relying on Q-values. The DDPG algorithm extends these ideas by combining the value function learning of Q-learning with policy optimization, making it suitable for continuous action spaces. Unlike stochastic policy-based methods, DDPG learns a deterministic policy function $a=\pi_{\theta}(s)$ that maps each multivariate time series state directly to a continuous action, thus avoiding an expensive search for action selection.

Within the DDPG actor-critic framework, an actor network $\pi_{\theta}(s)$ outputs portfolio weights and a critic network $Q_{\phi}(s,a)$ evaluates them. The actor is then updated using the deterministic policy gradient:

[TABLE]

which ensures that selected actions maximize the critic’s Q-value. The critic is updated using Bellman’s equation, employing a soft‐updated target critic $Q_{\phi}(\tau=0.005)$ to enhance stability.

The agent adopts a TiDE-based architecture integrated within the DDPG framework. Historical features are passed to a two-layer TiDE encoder, which flattens the inputs and projects them into a latent representation using fully connected layers followed by residual blocks with ReLU activation and layer normalization. The TiDE actor maps the encoded state into a scalar action through a final linear transformation and a sigmoid activation, producing portfolio weights $\omega\in[0,1.0]$. The TiDE critic evaluates the quality of the state-action pair by concatenating the TiDE encoder’s output with the TiDE actor’s action and processes it through two fully connected layers with ReLU activations to output the Q-value. This encoder-based actor-critic structure enables the model to handle multivariate time series inputs efficiently, while the deterministic policy of DDPG facilitates continuous portfolio weight selection, offering finer control and reduced computation compared to discrete action methods.

3.5 Dataset and Experiments

We used a dataset consisting of monthly time series on the market return rate $r_{t}$ (Mkt) and risk-free asset return rate $r_{f}$ (RF) from 1927 to 2019. We separated into an initial training period (1927–1957), a validation period (1958–1988), and a test period (1989–2019). Daily returns are retrieved to compute the monthly realized volatilities. The original data set library is available on the Fama/French website.333Data in Fama/French 3 Factors csv, see: https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html. There are 11 macroeconomic and financial predictors for market environment indicators, shown in Table 1. An additional variable is the one-month lagged excess return. Lastly, one-to-three-month lags of an enhanced measure of the payout yield444Data in zipped csv, for paper [8], available here: https://sites.google.com/view/agoyal145. and one-to-three-month lags of the realized squared monthly volatilities are included. At each timestep, the agent can observe the 18 features including the current month and previous 11 months, as well as the logarithmic returns of the past 12 months, before making a decision. Using these data, experiments were conducted to compare three models: (i) Q-learning with a discrete state space and a discrete action space, both without leverage, $\omega\in\{0.0,0.1,\ldots,1.0\}$, and with 50% leverage available, $\omega\in\{0.0,0.15,\ldots,1.5\}$; (ii) DDPG with TiDE with a continuous action space, both without leverage, $\omega\in[0.0,1.0]$, and with 50% leverage available, $\omega\in[0.0,1.5]$; and a (iii) Buy-and-hold strategy with constant action $\omega=1.0$. We introduce buy-and-hold as a comparison baseline to determine whether active investment approaches can outperform a passive investment policy. We also introduce the opportunity for Q-learning and DDPG-TiDE strategies to use leverage (i.e., to select values $w>1$) to see if they can learn strategies that profit from borrowing to invest.

The Q-learning and DDPG algorithms share a common dataset and trading environment module, although they differ in their state and action representations. Q-learning utilizes a discrete state and action space, achieved through K-means clustering and fixed weight increments, while DDPG operates in continuous spaces, employing a multi-step neural network time-series training approach. Furthermore, DDPG incorporates Ornstein–Uhlenbeck (OU) noise to facilitate exploration, as its policy network is inherently deterministic. A multi-step replay buffer, which stores transitions ($s,a,r,s^{\prime}$), stabilizes training by breaking correlations in consecutive data and allowing mini-batch updates. The TiDE architecture in DDPG processes the time-series state before the agent encounters data: an encoder captures temporal features, while actor and critic networks propose actions and evaluate Q-values, respectively. Grid-search is used to explore the candidate range of hyperparameters.

The performance of each algorithm, optimized with its respective hyperparameters, is evaluated once on the test set and quantified using three performance metrics (see Table 2): (i) Logarithmic utility maximizes the utility function when the reward is set as log-return by continuously adding up the rewards, that is, optimizing the final wealth under the Kelly criterion; (ii) Portfolio value measures the cumulative returns over a given interval, with $W_{0}=1$ as the initial investment and $W_{t}$ as the total portfolio value at time $t$; and (iii) Sharpe ratio is defined as the average excess return of a portfolio over the standard deviation of the excess portfolio return, and is calculated over a 12-month rolling window.

4 Results and Discussion

In this section, we present results during the test period (1989–2019). As expected, we observe that the cumulative rewards (or logarithmic utility) and the total portfolio value (or total wealth, or cumulative returns) for each strategy follow identical trends, as the reward is simply a logarithmic transformation of wealth. Therefore, to avoid duplication, we do not present or discuss cumulative rewards.

In Fig. 1, we see that Q-learning strategies tend to generate a steady, shallow increase in portfolio value over long periods of time, particularly before 2008. The final portfolio value, or wealth, for Q-learning, both without leverage (3.88) and with 50% leverage available (4.65), is much lower than the buy-and-hold strategy (17.77). In contrast, the portfolio values of the DDPG-TiDE strategies closely follow the value of buy-and-hold. The final portfolio value for DDPG-TiDE without leverage (14.85) is slightly below buy-and-hold, and the final portfolio value for DDPG-TiDE with 50% leverage available (21.06) is slightly above buy-and-hold. These trends indicate that Q-learning strategies largely follow a risk-averse strategy, by predominately investing in the risk-free asset; while DDPG-TiDE strategies largely follow a risky strategy, by predominately investing in the market index.

Fig. 2 plots the Sharpe ratio of each strategy over a rolling 12-month window. For Q-learning strategies, which tend to invest in the risk-free asset, Sharpe ratios are much lower on average and are zero for long periods (e.g., 1992–2007). In contrast, Sharpe ratios for DDPG-TiDE strategies closely track buy-and-hold, with average Sharpe ratio of 1.13 when no leverage is available, and average Sharpe ratio of 0.99 when leverage is available. Both of these values are slightly higher than the average Sharpe ratio for buy-and-hold (0.95), indicating that active DDPG-TiDE strategies generate slightly better risk-adjusted returns than a passive buy-and-hold index tracker. Furthermore, we note that while the use of leverage allows DDPG-TiDE to accumulate the highest returns (see Fig. 1), the increased risk of using leverage results in a slightly lower average Sharpe ratio.

We can concretely understand the performance of each strategy by observing the action taken by each strategy at each timestep. Fig. 3 shows the portfolio weight assigned to the risky asset each month, such that when $w=1$ all investments are held in the market index and when $w=0$ all investments are held in the risk-free asset. For values $w>1$, the strategy leverages capital by borrowing to invest more than current total wealth; for example, a value $w=1.5$ would imply 50% leverage such that 1.5 times current wealth is invested in the risky asset. We see that the Q-learning strategy invests fully in the risk-free asset for long periods (e.g., 1992–2007), which coincides with the period of zero Sharpe ratio shown in Fig. 2, and the smoothly increasing returns with shallow gradient shown in Fig. 1. In contrast, during this same period, DDPG-TiDE often invests fully in the risky asset, leading to a more erratic Sharpe ratio that correlates with buy-and-hold (Fig. 2). Overall, this explains why the wealth of DDPG-TiDE largely tracks buy-and-hold, and also explains how DDPG-TiDE uses leverage (often taking values $w=1.5$) to take advantage of the bull market period post 2008 to outstrip the returns of buy-and-hold (see Fig. 1).

5 Conclusions

We have explored the use of RL as a model-free approach to optimize portfolio allocation between a risk-free asset and a market index. The novel contribution of this work is the introduction of a TiDE-based architecture into the DDPG RL framework. We showed that a continuous-action DDPG-TiDE strategy outperforms a simple discrete-action Q-learning strategy in all metrics; it is also able to generate more profits than buy-and-hold by applying leverage during bull periods; and it generates the highest average Sharpe ratio. These results suggest that the integration of TiDE within a DDPG RL framework to tackle the optimal asset allocation problem is a fruitful avenue for further exploration.

We have deliberately used a simplistic simulation framework. To advance this line of research, future work will incorporate a wider range of RL frameworks and methodologies. In addition, practical trading costs will be taken into account, and penalty mechanisms will be introduced in both algorithm design and code implementation to ensure the practicality and generalizability of the strategy. Our long-term goal is to use a multi-agent modeling approach (e.g., [7]) to develop intelligent collectives of reinforcement learning agents that offer user-specific personalized investment strategies (for details, refer to the AI for Collective Intelligence research strategy [4]).

Acknowledgements

This work was supported by UKRI EPSRC Grant No. EP/Y028392/1: AI for Collective Intelligence (AI4CI).