Avoiding Backtesting Overfitting by Covariance-Penalties: an empirical investigation of the ordinary and total least squares cases

TL;DR

This paper introduces a Covariance-Penalty correction method to prevent backtesting overfitting in trading strategies, demonstrating its effectiveness through empirical tests on over 1300 assets, with Total Least Squares outperforming Ordinary Least Squares.

Contribution

It proposes a novel Covariance-Penalty correction approach for avoiding backtesting overfitting in financial strategies, validated through extensive empirical analysis.

Findings

Covariance-Penalties effectively reduce backtesting overfitting.

Total Least Squares outperforms Ordinary Least Squares in strategy performance.

The proposed method is validated on over 1300 assets.

Abstract

Systematic trading strategies are rule-based procedures which choose portfolios and allocate assets. In order to attain certain desired return profiles, quantitative strategists must determine a large array of trading parameters. Backtesting, the attempt to identify the appropriate parameters using historical data available, has been highly criticized due to the abundance of misleading results. Hence, there is an increasing interest in devising procedures for the assessment and comparison of strategies, that is, devising schemes for preventing what is known as backtesting overfitting. So far, many financial researchers have proposed different ways to tackle this problem that can be broadly categorised in three types: Data Snooping, Overestimated Performance, and Cross-Validation Evaluation. In this paper, we propose a new approach to dealing with financial overfitting, a Covariance-Penalty Correction, in which a risk metric is lowered given the number of parameters and data used to underpins a trading strategy. We outlined the foundation and main results behind the Covariance-Penalty correction for trading strategies. After that, we pursue an empirical investigation, comparing its performance with some other approaches in the realm of Covariance-Penalties across more than 1300 assets, using Ordinary and Total Least Squares. Our results suggest that Covariance-Penalties are a suitable procedure to avoid Backtesting Overfitting, and Total Least Squares provides superior performance when compared to Ordinary Least Squares.

Figures

Tables

Table 1. Table 1: Categorisation of all the four lines of investigations in backtesting overfitting.

Breadth / Solution	Multiple Hypothesis	In-Out Sample Assessment
Generic	Data Snooping	Cross-Validation
Assumptions	Overestimated Performance	Covariance-Penalty

Table 2. Table 2: Aggregated statistics of the assets used during our empirical evaluation.

Asset Pool*	N	Avg Return	Vol	Sharpe Ratio	Calmar Ratio	Monthly Skewness	VaR 95%
All Assets	1361	0.0173	0.0425	0.4064	0.0809	-0.9301	-0.9314
World Equity Indices	18	0.0152	0.0717	0.2114	0.0504	-0.8170	-1.0048
S&P, SX5E, FTSE, DJIA, Russel and Nikkei Equities	1289	0.0172	0.0439	0.3915	0.0796	-0.8816	-0.9316
World Swaps Indices	14	-0.0197	0.0624	-0.3149	-0.0433	-0.1039	-0.9955
Rates Aggregate Indices	16	0.1220	0.0637	1.9135	0.5504	-0.8227	-0.9440
World Currencies	24	-0.0025	0.0315	-0.0798	-0.0157	-1.0052	-0.8856

Table 3. Table 3: Statistical comparisons between different Covariance-Penalty approaches and the baseline (Naive), using paired t-test and Bonferroni Correction for multiple hypothesis test. The boldfaced pairs are the ones not significantly different at 5% after Bonferroni correction.

Val Method		ROS Sharpe ratio		p-value
Left	Right	Mean diff	t-stat	Value	Bonferroni
OLS-AIC	OLS-Imp SR	-0.0235	-3.7573	0.00018	0.00376
OLS-AIC	OLS-Naive	0.0575	10.061	0.00001	0.00001
OLS-AIC	OLS-R-Squared	-0.05351	-8.6793	0.00001	0.00001
OLS-AIC	TLS-Imp SR	-0.1036	-12.0062	0.00001	0.00001
OLS-AIC	TLS-Naive	0.03583	3.7806	0.000163	0.003430
OLS-AIC	TLS-R-Squared	-0.1048	-12.1588	0.00001	0.00001
OLS-Imp SR	OLS-Naive	0.08118	13.8264	0.00001	0.00001
OLS-Imp SR	OLS-R-Squared	-0.0299	-4.3723	0.00002	0.00028
OLS-Imp SR	TLS-Imp SR	-0.0800	-9.4425	0.00001	0.00001
OLS-Imp SR	TLS-Naive	0.05942	6.2843	0.00001	0.00001
OLS-Imp SR	TLS-R-Squared	-0.08122	-9.5304	0.00001	0.00001
OLS-Naive	OLS-R-Squared	-0.1111	-14.7347	0.00001	0.00001
OLS-Naive	TLS-Imp SR	-0.1612	-17.9283	0.00001	0.00001
OLS-Naive	TLS-Naive	-0.02176	-2.4326	0.01512	0.31753
OLS-Naive	TLS-R-Squared	-0.1624	-17.9653	0.00001	0.00001
OLS-R-Squared	TLS-Imp SR	-0.0501	-6.3812	0.00001	0.00001
OLS-R-Squared	TLS-Naive	0.08934	9.0067	0.00001	0.00001
OLS-R-Squared	TLS-R-Squared	-0.05129	-6.5487	0.00001	0.00001
TLS-Imp SR	TLS-Naive	0.1394	14.8044	0.00001	0.00001
TLS-Imp SR	TLS-R-Squared	-0.00118	-0.4581	0.64694	1.00000
TLS-Naive	TLS-R-Squared	-0.1406	-14.7307	0.00001	0.00001

Equations

$$t_{score}=\frac{\bar{R}}{{\sigma }_R}\times \sqrt{T}=\widehat{SR}\times \sqrt{T}$$

$$p^S=Pr(|t|>t_{score})$$

$$p^M=Pr(max(|t_i|,i=1,...,n)>t_{score})=$$

$$=1-\prod _{i=1}^{n}Pr(|t_i|\le t_{score})=1-(1-p^S{)}^n$$

$$p^M=Pr(|t|>\widehat{HSR}\times \sqrt{T})$$

$$E[X_1{X}_2\cdots X_{2n}]=\sum _{i=1}^{2n}\prod _{i\ne j}E[X_i{X}_j]$$

$$E[X_1{X}_2\cdots X_{2n-1}]=0$$

$$E[xy]=$$

$$E[x^2{y}^2]=$$

$$E[x^3{y}^3]=$$

$$E[x^4{y}^4]=$$

$${\mu }_1=$$

$${\mu }_2=$$

$${\mu }_3=$$

$${\mu }_4=$$

$$SR=\frac{\rho }{\sqrt{1+{\rho }^2}}$$

$${\gamma }_3=\frac{2\rho (3+{\rho }^2)}{(1+{\rho }^2{)}^{\frac{3}{2}}},$$

$${\gamma }_4=\frac{3(3+14{\rho }^2+3{\rho }^4)}{(1+{\rho }^2{)}^2}$$

$$SR=\frac{{\mu }_1}{{\mu }_2^{1/2}}=\frac{{\sigma }_X{\sigma }_R\rho }{{\sigma }_X{\sigma }_R\sqrt{{\rho }^2+1}}=\frac{\rho }{\sqrt{{\rho }^2+1}}$$

$${\gamma }_3=\frac{{\mu }_3}{{\mu }_2^{3/2}}=\frac{2\rho (3+{\rho }^2)}{(1+{\rho }^2{)}^{3/2}}$$

$${\gamma }_4=\frac{{\mu }_4}{{\mu }_2^2}=\frac{3(3+14{\rho }^2+3{\rho }^4)}{(1+{\rho }^2{)}^2}$$

$${\hat{\beta }}^{OLS}=(Z^{\prime }Z{)}^{-1}{Z}^{\prime }R$$

$${\hat{\beta }}^{TLS}=(Z^{\prime }Z-{\sigma }_{k+1}^{2}I{)}^{-1}{Z}^{\prime }R$$

$$M^{OLS}=Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }$$

$$M^{TLS}=Z(Z^{\prime }Z-{\sigma }_{k+1}^{2}I{)}^{-1}{Z}^{\prime }$$

$$Tr(M^{TLS})\ge Tr(M^{OLS})$$

$$Tr(M^{TLS})=\sum _i\frac{{\lambda }_i^2}{({\lambda }_i^2-{\sigma }_{k+1}^2)}\ge k=Tr(M^{OLS})$$

$$E[Erro{r}_{out}^2]=E[|R-Xb{|}_{out}^2]$$

$$=\frac{1}{T}\sum |R_t-X_{t}b{|}_{in}^2-Cov({\mu }_t,\hat{\mu }{)}_t$$

$$={\widehat{sse}}^2-\frac{2{\sigma }_R^2}{N}Tr(M)$$

$$E[{\rho }_{out}^2]={\hat{\rho }}_{in}^2-\frac{2}{N}Tr(M)$$

$$R=MX$$

$$M=X(X^{\prime }X-{\sigma }_{n+1}^{2}I{)}^{-1}{X}^{\prime }$$

$$tr(M)=\sum _i\frac{{\lambda }_i^2}{({\lambda }_i^2-{\sigma }_{n+1}^2)}>n$$

$$E[S{R}_{out}]$$

Full text

Keywords: Algorithmic Trading, Overfitting, Covariance-Penalty, Total Least Squares, Ordinary Least Squares, Quantitative Finance

MSC Numbers: 60G10, 62E15, 62P05, 62F99, 91G70, 91G80

1 Introduction

Systematic trading strategies are rule-based procedures which choose portfolios and allocate assets, in order to deliver a set of returns which optimize some performance criterion, help to diversify a larger portfolio, or perhaps deliver returns specifically tailored to market conditions, such as providing insurance during rises of market volatility. In order to attain any of these desired return profiles, quantitative strategists must determine a large array of trading parameters, including determining the universe of assets being held, the appropriate trading horizons, the type of strategy being considered and many other possible parametrizations. Normally the analyst tries to identify the appropriate parameters by using all the historical data available, or by holding-out some of the most recent data in order to use it as an out-sample validation set. However, both backtesting strategies suffer from drawbacks that can render their findings misleading.

Partly due to the abundance of misleading results produced from backtests, and partly due to the endemic abuse of backtested results, there is an increasing interest in devising procedures for the assessment and comparison of strategies [22, 42, 3]. The goal is to devise schemes for preventing what is known as backtesting overfitting. Backtest overfitting can be understood as a collateral effect that occurs during the identification of strategies, resembling other practical modelling problems: overfitting in supervised learning [14, 11, 29], or data-snooping due to multiple hypothesis testing biases, (also known as p-hacking), a problem occuring frequently in the applied sciences [25, 6].

So far, many financial researchers have proposed different ways to tackle this problem that can be broadly categorised in three types: Data Snooping, Overestimated Performance, and Cross-Validation Evaluation. In summary, the Data Snooping and Overestimated Performance techniques aim to identify spurious results by taking into account the number of trials and the performance from all attempted strategies in a multiple hypothesis testing framework. Conversely, Cross-Validation approaches provide more practical and generic mechanisms to individually fine-tune trading strategies.

In this paper we propose a new approach to dealing with financial overfitting, a Covariance-Penalty Correction, in which a risk metric is lowered given the amount of parameters and data used to underpins a trading strategy. We outline the foundation and main results behind the Covariance-Penalty correction for trading strategies. After that, we pursue an empirical investigation, comparing its performance with some other approaches in the realm of Covariance-Penalties across more than 1300 assets, using Ordinary and Total Least Squares – with the last providing superior performance. Before delving into this novel perspective, we first briefly exhibit a historical context about the available methodologies.

2 Literature Review

We can outline, chronologically, three distinct approaches in the literature to evaluate and deal with backtesting overfitting: Data Snooping, Overestimated Performance, and Cross-Validation Evaluation. The problem of overfitting cannot be understated, and innumerable references highlight the issues with p-hacking which has been an issue for considerable periods but making headlines more recently (see e.g., [35], [26], [7], [13]), and [24]) although it may not always be willful ([16]), and is pernicious in finance (see for instance, [44]).

2.1 Data Snooping

Data snooping is a problem commonly encountered in the realm of econometrics, specifically when a researcher is exploring asset pricing models. Among several definitions in different contexts, in finance it is determined by the act of finding significant yet spurious risk premia factors, due to the unavoidable reuse of the same time series [8, 48]. The first work in this direction can be attributed to Lo and MacKinlay [33] attempted to highlight and quantified the effect of data snooping in risk premia studies.

More precisely, the data-snooping problems/solutions tend to focus on the issue of Multiple Hypothesis Testing. Standard methods to adjust for snooping from multiple hypothesis testing biases, such as by Bonferroni and Holm to control family wise error rates (FWER111Probability of making one false discovery, that is, non-rejecting a strategy with Sharpe Ratio $=0$.) and by Benjamini-Hochberg-Yekutieli (BHY) for controlling False Discovery Rates (FDR222Expected frequency of non-rejecting a strategy with Sharpe Ratio $=0$.) are in relatively common use in applied sciences [25, 6]. These methods adjust for the likelihood of false discoveries due to the correlation between multiple tests and are an effective way to prevent what is otherwise known as p-hacking in the applied sciences. Nonetheless, since they adjust to effectively worst-case scenarios, they are known to result in significant reductions of power.

Due to the low power of the FWER adjustments, Westfall and Young [47] introduced bootstrap methods to estimate the empirical correlation between hypothesis tests as a way of optimising the power of the resulting adjusted test statistics. Following on these, White [48] outlined one of the first attempts to improve test statistics by applying bootstrap methods to identify spurious relationships in the realm of trading strategies, applying them to technical trading rules [45]. Romano and Wolf [40] advanced in procedures to more adequately correct multiple testing and resourcing in more adequate forms of bootstrapping for standard errors calculations; we point also to more recent works in this direction [41, 39, 42, 23].

Since Romano and Wolf can be seen as an extension of White’s work, we briefly outline its algorithm for data snooping diagnosis. Let $r_{it}=R_{it}-R_{bench,t}$ represent the excess return ($R$) of the asset $i$ in relation to a benchmark $bench$ at time $t$. Each asset can represent a trading strategy with different parametrisation, and the benchmark can be the usual 3-month London Interbank Overnight Rate return or simply an offset. Given a differentiable performance function $P$, a significance level $\alpha$, and a number of bootstrap samples $B$ ($b=1,...,B$), do:

• •

Compute each asset performance: $p_{i}=P(r_{i1},...,r_{it},...,r_{iT})$.

• •

Set $\hat{V}=\sqrt{T}*\max(p_{1},...,p_{n})$ as the maximum estimated performance across the available assets.

• •

Set $k=1$, and while all assets have not been rejected:

For $b=1,...,B$ bootstrap samples, do:

1. (a)

Take a bootstrap sample $r_{.1}^{(b)},...,r_{.T}^{(b)}$ from the sequence $r_{.1},...,r_{.T}$. The . notation is the abbreviation for ”all assets”. 2. (b)

Compute each asset performance: $p_{i}^{(b)}=P(r_{i1}^{(b)},...,r_{it}^{(b)},...,r_{iT}^{(b)})$ 3. (c)

Define the statistic $V_{b}=\sqrt{T}*\max(p_{i}^{(b)}-p_{1},...,p_{n}^{(b)}-p_{n})$ 2. 2.

Find the critical value $c_{k}$ by estimating the quantile $(1-\alpha/k)$% of $V_{b}$ empirical distribution. 3. 3.

Reject all strategies that $(\sqrt{T}*p_{i})>c_{k}$; remove these and place them in the echelon $k$. 4. 4.

$k+=1$

• •

Report all echelons to the user.

Each echelon represent a subset of trading strategy that tend to perform similarly, but are superior/inferior to the followed/previous iteration. As it can be seen, this method is based on a stepdown approach, since we start with the null hypothesis that correspond to the largest test statistic. A final comment: for the first iteration it is possible to calculate the p-value for the Bootstrap Reality Check [48] by: $p_{BRC}=\frac{\sum_{b=1}^{B}V_{b}>\hat{V}}{B}$. This statistic addresses the question whether the strategy that appears best in the observed data really beats the benchmark; it does not attempt to identify as many outperforming strategies as possible. We also highlight that this presentation took the non-studentised version of Romano and Wolf procedure; for the studentised version, which provide some improvements to the statistical procedure, refer to [40].

Finally, another related approached called Model Confidence Set [21, 20], acknowledges the limitations of the data, such that uninformative data yield a MCS with many models, whereas informative data yield a MCS with only a few models. By pairwise comparisons, this methodology creates confidence sets (analogous to confidence intervals) that contains the best model or models given a specified significance level.

2.2 Overestimated Performance

In Harvey and Liu [22] the authors exhibit a multiple hypothesis testing framework devoted to identify strategies incorrectly classified as profitable. Before delving into the [22] work, it should be mentioned that similar works have also proposed ways to incorporate uncertainty and perform adjustments in the strategies performance (mainly for Sharpe Ratios). A section of these tends to favour hypothesis tests and confidence interval computations, starting from: (i) the classical i.i.d. Gaussian returns [28]; (ii) then dropping Gaussian assumption with the support of (independent) bootstrap [46]; (iii) more ahead being able to pinpoint an asymptotic distribution for the Sharpe Ratio under non-i.i.d and more mild assumptions [32, 37]; and finally (iv) dropping i.i.d. and underlying distribution assumptions by applying improved forms of bootstrapping [30]. Another approach is by adjusting the Sharpe Ratio by computing the so-called Probabilistic Sharpe Ratio [4]. This measure computes the probability that a strategy performance is greater than a benchmark, taking into account the skewness and kurtosis of the returns.

However, all these works do not take into account the number of trials did to fully develop the strategy, or when a hypothesis test is being made to compare different strategies. This information is relevant, because some improvement is expected whenever the analyst have access to information coming from the backtest: increasing overall performance, removing drawdowns, etc. In this way, he can design stop/loss rules, change leveraging rules, with undoubtly impact in the strategy performance when the backtest is remade. The authors [22] treat this as a Multiple Hypothesis Testing situation [25, 6]. To illustrate this concept, if we suppose that the (excess) returns are i.i.d. and follows a Normal distribution, then the Sharpe Ratio can be reinterpreted as t-scores:

[TABLE]

where $\hat{SR}$ is the estimated Sharpe Ratio and $T$ is the backtest length. We can compute a t-test using this $t_{score}$, aiming to check the probability for observing a departure from the null hypothesis ($H_{0}:SR=0$) as extreme as the difference present in our sample data. This is define as:

[TABLE]

in which $t$ denotes a random variable that follows a t-distribution with $T-1$ degrees of freedom, and $p^{S}$ is commonly dubbed as p-value, which in this case is computed for a single hypothesis ($S$). If a research have explored several strategies, then some overstatement will appear in the $\hat{SR}$, and we could interpret this as the researcher did tacitly multiple attempts to obtain such result. As a initial estimation of this overestatement, suppose that all attempts are independent (unreal, but we drop this more ahead), then we can rewrite 2 as:

[TABLE]

by following [22]: when $n=1$ (single test) and $p^{S}=0.05$, $p^{M}=0.05$, so there is no multiple testing adjustment. If $n=10$ and we observe a strategy with $p^{S}=0.05$, $p^{M}=0.401$, implying a probability of about 40% in finding an investment strategy that generates a $SR$ that is at least as large as the observed $\hat{SR}$, but much larger than the 5% probability for a single test. It is clear that multiple testing greatly reduces the statistical significance of single test. Hence, $p^{M}$ is the adjusted p-value, that reflects the likelihood of finding a strategy that is at least as profitable as the observed strategy after $n$ attempts. By equating the p-value of a single test to $p^{M}$, we obtain the defining equation for the multiple testing adjusted (haircut) Sharpe ratio $\hat{HSR}$:

[TABLE]

since $p^{M}$ is larger than $p^{S}$, $\hat{HSR}$ will be smaller than $\hat{SR}$. However, when the test statistics are dependent, however, the approach in the example is no longer applicable as $p^{M}$ generally depends on the joint distribution of the $n$ test statistics. For this more realistic case, the authors apply techniques that controls the FWER and FDR. These methods are Bonferroni and Holm p-value corrections to control FWER [10] and BHY procedure for FDR [6]. Below, in Figure 1 we display some charts with pre and post-adjustment of Sharpe Ratios, as well as the Haircuts333In order to facilitate researches interested on using this approach, they provide a link for the publicly available Matlab code (http://faculty/fuqua.duke.edu/~charvey/backtesting). It is also possible to apply the same procedure for the Value-at-Risk metric (and therefore, making the same assumptions – check endnote 23 from [22]).:

As it can be seen, the Haircut is a non-linear function and is highly affected by the number of trials, and the method of adjustment. The authors advocate by the BHY method in financial applications, as they argue that it is more reasonable to control for the rate of false discoveries, rather than controlling for not making any false discovery. Finally, as the authors points out, the main caveat of this method is that all the analysis is based with in-sample data, and as they observe a hybrid strategy might be useful as a double validation of the selected trading strategy.

In line with multiple testing literature, there remain challenges associated with knowing which analyst observations of data are, in fact, uses of the data and thus, tests. Other challenges of multiple testing (in particular, when using brute-force penalisations such as BHY, Holm and Bonferroni) include:

• •

What is the impact of common knowledge?

• •

How does prior information impact hypotheses being tested? (and strategies being designed)?

• •

If multiple analysts look at the same data, should they be penalized based on each other observations?

• •

If the tests were conducted in a different order, the results would be different, so how does this impact the philosophy of the approach?

• •

How do we know if we have been influenced by others who have seen the same data?

• •

Is it even possible to have no knowledge of relevant financial history?

While care and concern over the overuse of history is imperative, it is not altogether clear whether the philosophy of multiple testing, especially as done via BHY, Holm and Bonferroni, is reasonable for financial applications. Nonetheless, we believe the methodology is absolutely informative and can be used in conjunction with our own. As yet an unresolved area, preventing overfitting requires a multiplicity of approaches.

2.3 Cross-Validation Evaluation

In Bailey et al. [3] the authors provide a set of definitions, tools and benchmarks to characterize the so-called backtest overfitting. In certain sense this work extend previous research and findings from the same group of authors, such as the probability that a Sharpe Ratio is inflated [4] and to determine the minimum track record for a Sharpe Ratio to be statistically different from a given threshold [2].

In terms of definitions, the authors establish that a strategy selection overfits if the expected performance in-sample is less than the median performance rank out-of-sample of all strategies. By strategy selection, it means a way of ranking the available strategies using a set of performance metrics (e.g., Sharpe Ratio).

In relation to tools they propose the procedure called Combinatorially-Symmetric Cross-Validation (CSCV), which is briefly outlined below and represented in Figure 2.

Split the profit and loss of several strategies in $S$ even size blocks over the backtested horizon (where $S$ is a even natural number); 2. 2.

Reshuffle these blocks in all feasible arrangements such that the natural ordering is as most preserved as possible; and 3. 3.

For each first half (in-sample) of the arrangement, find the optimal strategy for a given performance measure and compare it to its own ranking in the out-of-sample set (second half).

With these rankings and comparisons, the authors compute four complementary metrics:

Probability of Backtest Overfitting: the probability that the model configuration selected as optimal in-sample will underperform the median of the $N$ models configurations out-of-sample. 2. 2.

Performance degradation: this determines to what extent greater performance in-sample leads to lower performance out-of-sample (discussed in [2]). 3. 3.

Probability of loss: the probability that the model selected as optimal IS will deliver a loss OOS. 4. 4.

Stochastic dominance: this analysis determines whether the procedure used to select a strategy IS is preferable to randomly choosing one model configuration among the $N$ alternatives.

Also, in terms of benchmarks the authors propose some synthetic and practical test cases. In this sense, the objective of these studies is to assess if the proposed framework is able to dodge from overfitted strategies with different set of hyperparameters (number of strategies, backtest length and overfit degree).

We can highlight two main critics assessment towards this work:

Preservation of temporal dependence: the authors claim that their method preserves the temporal dependence. In their support are the fact that their method reorders the blocks in-sample and out-of-sample and someone might argue that daily returns tends to present weak autocorrelation. However, this claim about preservation of time dependence is clearly not absolutely true: just check the third to fifth row at Figure 2. 2. 2.

High overlap degree in the in-sample and out-of-sample metrics: it can be easily noticed that there is a high level of juxtaposition of blocks appearing several times in the out-of-sample part (and similarly in the in-sample). The authors in proposing the CSCV, using as inspiration the k-fold cross-validation, forgot to analyse deeper the impact of the highly correlated out-of-sample folders (something that do not occur in the case for the k-fold cross-validation). This correlation exerts an impact in the computation of standard errors and hypothesis tests [29, 5], and therefore decrease the out-of-sample assessment quality.

2.4 Summary

Table 1 present a categorisation of all the previous lines of investigation – with the addition of Covariance-Penalty –, in terms of their breadth/generality of use in practice and the type of solution used to tackle backtesting overfitting.

The Data Snooping and Overestimated Performance authors identify that the root cause of any overfitting or spurious results is due to the multiple trials (hypothesis) employed by the analyst during the strategies fine-tuning step. However, the Data Snooping solutions tend to be more generic – it is based on bootstrapping returns –, with the expense of a increased computational burden. The Overestimated Performance solutions are based on a set of assumptions, allowing the authors to find closed-form solutions or simpler algorithms that make the performance correction aspect easier to compute.

In contrast, the Cross-Validation Evaluation and Covariance-Penalty Correction are focused on avoiding the apparent in-sample performance, by uncovering the expected generalisation (out-sample) potential of a strategy. Similarly, they divert in terms of their amplitude: the Covariance-Penalty Correction methods require some assumptions on the joint behaviour of the asset and trading strategy, which is not the case for the majority of the Cross-Validation procedures. However, this lack of assumptions demand an extra computational processing in respect to the Covariance-Penalty approach.

3 Covariance-Penalty for Trading Strategies

This section outline the theoretical results and notations used along this paper. We start by enunciating single period linear strategies, and some of its properties. Then we move to optimisation of trading strategies by maximising correlation, and its links to Sharpe ratio utility function. This result/link allows us to conclude this section with the main result of this paper: Covariance-Penalty correction of Sharpe ratio.

3.1 Single period linear strategies

We consider the (log) returns of a single asset, ${R_{t}\sim\mathscr{N}(0,\sigma_{R}^{2})}$ returns with auto-covariance function at lag $k$, $\gamma(k)=E[R_{t}R_{t-k}]$, together with corresponding auto-correlation function (ACF), $c(k)=\gamma(k)/\gamma(0)$ at lag $k$.

Our main aim is to work with strategies based on linear portfolio weights (or signals) $X_{t}=\Sigma_{1}^{\infty}a_{k}R_{t-k}$ for coefficients $a_{k}$ generating the corresponding dynamic strategy returns $S_{t}=X_{t}\cdot R_{t}$ (here, and always, the signal, $X_{t}$ is assumed to only have appropriately lagged information). Example strategy weights include exponentially weighted moving averages $a_{k}\propto\lambda^{k}$, simple moving averages $a_{k}=\frac{1}{T}\mathbbm{1}_{[1,\ldots,T]}$, forecasts from ARMA models, etc. Most importantly, the portfolio weights $X$ are normal and jointly normal with returns $R$.

We restrict our attention to return distributions over a single period. In the case of many momentum strategies, this period can be one day, if not longer. For higher-frequency intra-day strategies, this period can be much shorter. The pertinent concern is that the horizon (i.e., one period) is the same horizon over which the rebalancing of strategy weights is done. If weights are rebalanced every five minutes, then the single period should be five minutes. This is a necessary assumption in order to ensure the joint normality of (as yet indeterminate) signals and future returns. Moreover, this assumption will give some context to our results, which imply a maximal Sharpe ratio, maximal skewness and maximal kurtosis for dynamic linear strategies.

We are interested in characterizing the moments of the strategy’s unconditional returns, the corresponding standard errors on estimated quantities, and means of optimising various non-dimensional measures of returns such as the Sharpe ratio via the use of non-linear transformations of signals. Our goal is to look at unconditional properties of the strategy. It is important to avoid foresight in strategy design and this directly impacts the conditional properties of strategies (e.g., conditional densities involve conditioning on the currently observed signal to determine properties of the returns, which are just Gaussian). In the context of our study, we are concerned with one-period ahead returns of the unconditional returns distribution of our strategy, where both the signals and the returns are unobserved, and the resulting distributions (in our case, the product of two normals) are much richer and more realistic.

3.2 Properties of linear strategies

Given the joint normality of the signal and the returns, we can explicitly characterise the one-period strategy returns (see [9]). To allow for greater extendibility, we prefer to only consider the moments of the resulting distributions. These can be characterized easily using Isserlis’ theorem [27], which gives all moments for any multivariate normal random variable in terms of the mean and variance. We also refer to [19] who meticulously produces both non-central and central moments for powers and products of Gaussians. While this is a routine application of Isserlis’ theorem, the algebra can be tedious, so we quote the results.

Theorem 1** (Isserlis (1918) [27]).**

If $X\sim\mathscr{N}(0,\Sigma)$,then

[TABLE]

and

[TABLE]

where the $\sum\prod$ is over all the $(2n)!/(2^{n}n!)$ unique partitions of $X_{1},X_{2},\ldots X_{2n}$ into pairs $X_{i}X_{j}$.

Haldane’s paper quotes a large number of moment-based results for various powers of each normal. We quote the relevant results.

Theorem 2** (Haldane (1942) [19]).**

If $x,y\sim\mathscr{N}(0,1)$ with correlation $\rho$ then

[TABLE]

and thus the central moments of $xy$ are

[TABLE]

From these one period moments, (and a simple scaling argument giving the dependence on $\sigma(x)$ and $\sigma(y)$) we can characterise Sharpe ratio, skewness, etc., and can also define objective functions in order to determine some sense of optimality for a given strategy.

Theorem 3** (Linear Gaussian).**

For single asset returns and a one period strategy, $R_{t}\sim\mathscr{N}(0,\sigma_{R}^{2})$ and $X_{t}\sim\mathscr{N}(0,\sigma_{X}^{2})$ jointly normal with correlation $\rho$, the Sharpe ratio is given by

[TABLE]

the skewness is given as

[TABLE]

and the kurtosis is given by

[TABLE]

In the appendix, we extend equations (5) and (6) to the case of non-zero means.

Proof.

A simple application of Theorem 2 give us the following first two moments for our strategy $S_{t}=X_{t}\cdot R_{t}$: $\mu_{1}=E[S_{t}]=E[X\cdot R]=\sigma_{X}\sigma_{R}\rho$. and $\mu_{2}=Var[S_{t}]=\sigma_{X}^{2}\sigma_{R}^{2}(\rho^{2}+1)$ . Thus we can derive the following results for the Sharpe ratio,

[TABLE]

Moreover, we can see that the skewness,

[TABLE]

Finally, the kurtosis is given by

[TABLE]

∎

If we restrict our attention to positive correlations, all three dimensionless statistics are monotonically increasing in $\rho$. Consequently, strategies that maximize one of these statistics will maximize the others, although the impact of correlation upon Sharpe ratio, skewness and kurtosis is different. We illustrate the cross-dependencies in the following charts, depicting the relationships between the variables. In figure 3, the shaded blue histograms correspond to correlation ranges ($\{[-1,-0.5],[-0.5,0],[0,0.5],\ [0.5,1]\}$). We note that a uniform distribution in correlations maps into a higher likelihood of extreme Sharpe ratios and an even higher likelihood of extreme skewness and kurtosis.

Skewness ranges in $[-2^{3/2},2^{3/2}]\approx[-2.8,2.8]$. Unlike the Sharpe ratio, Skewness’ dependence on correlation tends to flatten, so to achieve 90% peak skewness, one needs only achieve a 0.60 correlation, while for a 90% peak Sharpe, one needs a correlation of 0.85. Kurtosis is an even function and varies from a minimal value of 9 to a maximum of 15. In practice, correlations will largely be close to zero and the resulting skewness and kurtosis significantly smaller than the maximal values.

Although we analyse the moments of the strategy $S_{t}=X_{t}R_{t}$, the full product density is actually known in closed form (see appendix A, [9] and [36]). It is clear that the distribution of the strategy is leptokurtic even when it is not predictive (when the correlation is exactly zero, the strategy has a kurtosis of $9$). In the limit as $\rho\rightarrow 1$, the strategy’s density approaches that of a non-central $\chi^{2}$, an effective best-case density when considering the design of optimal linear dynamic strategies.

An optimised strategy with sufficient lags (and a means of ensuring parsimony) may be able to capture both mean-reversion and trend and result in yet higher correlations. Annualised Sharpe ratios of between 0.5-1.5 are most common (i.e., correlations of between 3% to 9%) for single asset strategies in this relatively low-frequency regime.

3.3 Optimisation: Maximal Correlation, Total least squares

Many algorithmic traders will explain how problematic strategy optimisation is, given the endless concerns of over-fitting, etc. Although these are a concern, the naïve use of strategies which are merely pulled out of thin air is equally problematic, where there is no explicit use of optimisation (and, in its place more eye-balling strategies or targeting Sharpe ratios rather loosely, effectively a somewhat loose mental optimisation exercise). Practical considerations abound and real-world returns are neither Gaussian nor stationary. We argue irrespectively that using optimisation and a well-specified utility function as a starting point is a means of preventing strategies from being just untested heuristics. Unlike most discretionary traders’ heuristics (or rules of thumb) which have their place as a means of dealing with uncertainty (see for example [17]), heuristic quantitative trading strategies run the risk of being entirely arbitrary, or are subject to a large number of human biases, in marked contrast to the monniker quantitative investment strategies.

Where optimisation is used, the most common optimisation method is to minimize the mean-squared error (MSE) of the forecast. Our results show that rather than to minimize the $\mathcal{L}^{2}$ norm between our signal and the forecast returns (or to maximize the likelihood), if the objective is to maximize the Sharpe ratio, we must maximize the correlation.

We can see in figures 4 and 5, a depiction of fits of strategies applied to S&P 500 using EWMA and HW filters for a variety of parameters. The relationship between MSE and Sharpe ratio is not monotone in MSE for the EWMA filter as we see in figure 4, while it is much closer to being linear in the case of the relationship between correlation and Sharpe. For the case of HW (with two parameters), in figure 5 any given MSE can lead to a non-unique Sharpe ratio, sometimes with a very broad range, leading us to conclude that the optimization is poorly posed. The relationship of correlation to Sharpe is obviously closer to being linear, with higher correlations almost always leading to higher Sharpe ratios.

In the case of a one-dimensional forecasting problem with (unconstrained) linear signals, optimizing the correlation amounts to using what is known as total least squares regression (TLS) or orthogonal distance regression, a form of principal components regression (see, e.g., [18] and [34]). In the multivariate case, it would be more closely related to canonical correlation analysis (CCA).

Unlike OLS, where the dependent variable is assumed to be measured with error and the independent variables are assumed to be measured without error, in total least squares regression, both dependent and independent variables are assumed to be measured with error, and the objective function compensates for this by minimizing the sum squared of orthogonal distances to the fitted hyperplane. This is a simple form of errors-in-variables (EIV) regression and has been studied since the late 1870s, and is most closely related to principal components analysis. For $k$ regressors, the TLS fit will produce weights which are orthogonal to the first $k-1$ principal components.

So, if we consider the signal $X=Z\beta$ to be a linear combination of features, with $Z\in\mathbf{R}^{k}$ a $k$-dimensional feature space, then we note that

[TABLE]

but

[TABLE]

where $\sigma_{k+1}$ is the smallest singular value for the $T\times(k+1)$ dimensional matrix $\tilde{X}=[R,Z]$ (i.e., the concatenation of the features and the returns, see, e.g., [38]444A more common method for extracting TLS estimates is via a PCA of the concatenation matrix $\tilde{X}$, where $\hat{\beta}^{TLS}$ is chosen to cancel the least significant principal component).It is well known that, for the case of OLS, the smooth or hat matrix $\hat{R}=MR$ is given by

[TABLE]

with $Tr(M^{OLS})=k$, the number of features. In contrast,

[TABLE]

and effectively has a greater number of degrees of freedom than that of OLS, i.e.,

[TABLE]

with equality only when there is complete collinearity555In this case, it is also known that $Tr(M)=Tr(L)$ where $L=(Z^{\prime}Z-\sigma_{k+1}^{2}I)^{-1}Z^{\prime}Z$ and we know that the singular values of $\sigma(L)=\{{\lambda_{i}^{2}}/{(\lambda_{i}^{2}-\sigma_{k+1}^{2})}\}$ where $\lambda_{i}$ are the singular values of $Z$ (or correspondingly, $\lambda_{i}^{2}$ are the singular values of $Z^{\prime}Z$), and $\lambda_{1}\geq\cdots\geq\lambda_{k}>0$ ([31]). By the Wilkinson interlacing theorem, $\lambda_{k}\geq\sigma_{k+1}\geq 0$ (see [38]). Consequently,

$Tr(M^{TLS})=\sum_{i}\frac{\lambda_{i}^{2}}{(\lambda_{i}^{2}-\sigma_{k+1}^{2})}\geq k=Tr(M^{OLS})$

with equality iff $\sigma_{k+1}^{2}=0$ (i.e., when there the $R^{2}=100\%$ and consequently, OLS and TLS coincide). In other words, $Tr(M^{TLS})\geq Tr(M^{OLS})$.

For this reason, many people see TLS as an anti-regularisation method and may result in less-stable response to outliers (see for example, [49]). Consequently, there is extensive study of regularised TLS, typically using a weighted ridge-regression (or Tikhonov) penalty (see discussion in [49] for more detail on this large body of research).

While maximizing correlation rather than minimizing the MSE seems a very minor change in objective function, the formulas differ from those of standard OLS. The end result is a linear fit which takes into account the errors in the underlying conditioning information. We believe that it should be of relatively little consequence when the features are appropriately normalized, as is the case for univariate time-series estimation, although some authors have suggested that optimising TLS is not appropriate for prediction (see, e.g., [15] section 1.6.3). When we seek to maximize the Sharpe ratio of a strategy, the objective should not be prediction, but rather optimal weight choice.

3.4 Covariance-Penalty Correction

In order to partly address the issue of overfitting of dynamic models, we look to estimated objective functions, taking degrees of freedom into account. Most commonly used estimates include Akaike and Bayes’ information criteria, but these require the use of a likelihood. In our case, we do not have an exact likelihood, but instead choose to optimise the Sharpe ratio, and noting the monotonicity in $\rho$, this comes from maximizing $\hat{\rho}$. Consequently, we wish to consider the means of optimising correlation taking into account penalisations for degrees of freedom. This type of penalised optimum can be found via Mallows’ $C_{p}$ or via Cross validation.

We mirror the discussion in [12] on Covariance penalties, Section 12.3, which unifies the discussion of Mallow’s $C_{p}$, Stein’s Unbiased Risk Estimate (SURE), Akaike (AIC), and Bayesian Information Criteria (BIC). Each of these are methods for adjusting estimated in-sample goodness of fits measures, to produce better estimates of the out-of-sample performance. Effectively, they are a means of adjusting for the degrees of freedom of various models, akin to adjusted $R^{2}$ measures.

We are particularly interested in the biases which occur due to in-sample estimates of measures which we optimize. Standard errors will seem lower in-sample, due to the fact that we have simultaneously optimized correlation, and measured this optimum to produce an estimated Sharpe ratio . We would like to have a more sober estimate of the potential performance of our optimised strategies out of sample.

In our study, we do not fit probabilistic models or data-generating processes, but rather optimize the strategy. As a consequence, we do not have explicit MLE estimators (or Bayesian posterior), and no likelihood is explicitly used, so we cannot reproduce results for the AIC and the BIC. We are, however, able to find estimates for optimal $\hat{\rho}_{in}$ and the resulting out-of-sample adjustments for this $\hat{\rho}_{in}$ which give us a more reasonable expectation of the real $E[\rho_{out}]$ out of sample555To avoid confusion, we distinguish between our in-sample estimator $\hat{\rho}_{in}$ and the quantity we desire to estimate $\rho_{out}$ which would give us a better picture of the out-of-sample performance of our strategy.

Maximizing $\hat{\rho}_{in}$ is of course similar to performing a normalized $OLS$, and our interest would be in the analogue of adjustments to MSE. The first is easily found in Mallow’s $C_{p}$. Mallow’s formula states that

[TABLE]

where we have confined our attention to linear predictive models, i.e., $\hat{mu}=My$ where $M$ is the hat matrix or smooth matrix.

We recognise that $sse=(1-\rho^{2})\sigma_{R}^{2}$ from which we can easily derive the following:

Theorem 4** (Mallow’s $C_{p}$).**

[TABLE]

where

[TABLE]

with hat-matrix or smooth $M$

We do not prove this since it is a trivial extension of Mallow’s.

If $\hat{\rho}$ is estimated as a by-product of OLS, then $M=X(X^{\prime}X)^{-1}X$ (see [12], (§12.5)) and $Tr(M)=n$, where $k$ is the dimension of the column space of $X$ or the degrees of freedom for regressions (i.e. the total number of lags, where we assume that we have no problems with collinearity). The regularization term turns out to be $\frac{2\cdot k}{N}$ if $\hat{\rho}$ is estimated in-sample by OLS.

If, on the other hand, $\hat{\rho}$ is estimated to optimize the in-sample Sharpe ratio, i.e., through TLS, then it is well known that

[TABLE]

where $\sigma_{n+1}$ is the smallest singular value of the matrix $T\times(n+1)$ dimensional matrix: $\tilde{X}=[R,X]$ ([38]). As is well known (see for example, [49]) TLS is in fact an anti-regularisation method and may result in less-stable response to outliers.

In this case, it is also known that $Tr(M)=Tr(Z)$ where $Z=(X^{\prime}X-\sigma_{n+1}^{2}I)^{-1}X^{\prime}X$ and we know that the singular values of $\sigma(Z)=\{\frac{\lambda_{i}^{2}}{(\lambda_{i}^{2}-\sigma_{n+1}^{2})}\}$ where $\lambda_{i}$ are the singular values of $X$ (or correspondingly, $\lambda_{i}^{2}$ are the singular values of $X^{\prime}X$, $\lambda_{1}\geq\cdots\geq\lambda_{n}>0$ ([31]).

By the Wilkinson interlacing theorem, $\lambda_{n}\geq\sigma_{n+1}\geq 0$ (see [38]). Consequently,

[TABLE]

with equality iff $\sigma_{n+1}^{2}=0$ (i.e., when there the $R^{2}=100\%$ and consequently, OLS and TLS coincide).

Thus, while TLS can optimize the in-sample Sharpe-ratios over other predictive methods including OLS, the corresponding covariance penalization is larger than that of OLS and may result in an expected degradation of performance out-of-sample.444For AR models, $k$ is the autoregressive order. For $MA$ models, it is slightly different. For instance, if $M$ is the filter for an EWMA (i.e., an MA(1)), then the $tr(M)$ for the MLE estimate is in fact the normalisation constant. In other words, if $X=(1-\lambda)\sum\lambda^{k}X_{t-k}$, then $Tr(M)=(1-\lambda)$ rather than the MA order (i.e., 1). This means that a wider window-size ($\lambda\nearrow 1$) corresponds to fewer effective d.o.f. In the case of the TLS estimated window length, the penalization would be $tr(M)>1$

The corresponding penalization term can also be applied directly (via first-order conditions or the so-called Delta-method) to removing estimation bias from Sharpe ratio estimates, i.e.,

[TABLE]

for all $|\widehat{SR}_{in}|<\frac{\sqrt{2}}{2}$.

We also can extend Stein’s Unbiased Regression Estimates (SURE), i.e.

Theorem 5**.**

SURE

[TABLE]

In both cases, the degrees of freedom are used to adjust down the estimated Sharpe ratios. Consequently, more highly parametrised models will generally have higher overoptimism. The penalisation terms help to adjust for this overoptimism.

4 Empirical Assessment

In our empirical investigation, the quantitative strategist is required to build a trading strategy based on lagged information of a certain asset returns. Since the amount of lagged information is not given, he needs to discover it from the given in-sample data. The most naive way (called Naive method in our experiments) is to try different lags and to pick the one that maximizes the Sharpe ratio. Another traditional approach is to select the amount of lags using an information criteria, such as the AIC (Akaike Information Criteria [1]) for the OLS case. Nonetheless, we can also apply the Covariance-Penalty methods proposed in the previous section: the Implied Sharpe ratio (Imp SR - eq 13) and the R-Squared (eq 4) methods.

In summary, our main hypothesis is:

• •

Can the proposed approaches, Imp SR and R-Squared,outperform AIC and Naive in terms of out-sample/realised Sharpe ratio?

In addition to this, we verified the level of alignment between the Expected/In-Sample and Realised/Out-sample Sharpe ratios. With that, our goal was to analyse if there was any additional benefit in terms of performance estimation that can be made by following a Covariance-Penalty approach. In the following subsections we present the data and methodology used to test such hypothesis, as well as the findings and hypothesis testing.

4.1 Experimental Description

4.1.1 Datasets

Table 2 present aggregated statistics associated to the datasets used, whilst Figure 6 present the cumulative returns per asset pool. We have considered three main asset classes during our evaluation: equities, currencies, and fixed income. The data was obtained from Bloomberg, with the full list of 1361 assets tickers and period available at https://www.dropbox.com/s/oogzw8kaysx2qbp/data_list.csv?dl=0.

4.1.2 Model Evaluation Scheme

We have established a testing procedure to assess different Covariance-Penalties methodologies. Figure 7 summarize the whole process.

The process start by splitting a sequence of returns $r_{1},...,r_{T}$ in several in-sample ($IS$) and out-sample ($OS$) sets. The first pair, $IS_{1}$ and $OS_{1}$, are determined by the trading horizon $h$ and the first $IS$ batch size $I$. After that, the whole sequence of $IS_{k}$ and $OS_{k}$ are created by simply accumulating $IS_{k}=\{IS_{k-1},OS_{k-1}\}$ and rolling $OS_{k}=\{r_{I+(k-1)*h},...,r_{I+k*h}\}$ until $I+k*h\leq T$. Every Covariance-Penalty methodology – Naive, Implied SR, R-squared and AIC – tap into the $IS$ data to fine-tune a particular trading strategy. The measure of their success to avoid overfit is obtained when this strategy is applied in the $OS$ set. Using the sequence of in-sample and out-sample results we are able to compute performance metrics to evaluate and test the different hypothesis. We have used $h=21$ days and the initial period $I=1008$ days in our experiments; this process is repeated per dataset and Covariance-Penalty.

4.1.3 Trading Strategy and Performance Metrics

Below we present the steps taken to backtest the different trading strategies. For every pair of in-sample and out-sample data $(IS_{k},OS_{k})$, run:

• •

For every lag $p\in\{3,5,7,9,12,15,18,21,26,31,36,42,49,56,63,84,105,126\}$, do:

Fit the linear model using the in-sample set available ($IS_{k}$):

[TABLE]

with the coefficients $\beta_{0},\beta_{1},...$ identified via Ordinary or Total Least Squares. 2. 2.

Compute the risk of choosing $p$ using the in-sample set:

[TABLE]

with hat matrix $M$ and $P$ as a performance function, representing the different types of penalization used (AIC, Naive, Imp SR and R-squared). For the AIC and R-Squared it was also computed the expected Sharpe ratio ($\hat{SR}_{p}$).

• •

Pick $p^{*}=argmax_{p}\{L_{3},L_{5},...,L_{126}\}$, multiplying by $-1$ in the AIC case.

• •

Fit the linear model using $p^{*}$ and the in-sample set available ($IS_{k}$):

[TABLE]

• •

Test this model against the out-sample data ($OS_{k}$). Store out-sample realised strategy returns, $L_{p*}^{(k)}$ and $\hat{SR}_{p*}^{(k)}$.

With the sequence of results per asset in batches of in and out-sample, $\{(\hat{r}_{I},r_{I})$, $(\hat{r}_{I+1},r_{I+1}),...,$ $(\hat{r}_{I+h*k},r_{I+h*k});L_{p*}^{(k)};\hat{SR}_{p*}^{(k)}\}_{k=1}^{K-1}$, we computed some metrics related to the alignment between what was Expected and Realised:

• •

Correlation ($\rho$ as the Pearson correlation coefficient):

[TABLE]

• •

Mean Absolute Difference ($MAD$)

[TABLE]

where $SR^{(1:i)}$ is the Out-sample/Realised Sharpe ratio of a trading strategy until out-sample set $i$. Hence, the Correlation measures the association between the Expected and Realised Sharpe ratios, whilst $MAD$ is measuring its distance; the best case being $MAD=0$ and $Correlation=1$.

Finally, using the sequence as a whole per asset, $\{(\hat{r}_{I},r_{I})$, $(\hat{r}_{I+1},r_{I+1})$, $...,(\hat{r}_{I+h*(K-1)},$ $r_{I+h*(K-1)})\}$, we computed the Realised Sharpe Ratio. In all computations made, the Sharpe ratio was computed considering a 3-month Libor rate as the benchmark rate.

4.2 Results and Discussions

Figure 8 present the aggregated results for the different Covariance-Penalty methods, and types of Least Squares across the 1361 assets. In terms of Realised Out-Sample (ROS) Sharpe ratio, using some Covariance-Penalty method provides a boost in the unconditional average of 80-100% percent in relation to the Naive approach. Our two proposed approaches, Imp SR and R-squared, also improves 20-30% in relation to the AIC lag selection. Overall, this pattern is also visible in terms of alignment between Expected and Realised Sharpe ratios: MAD is reduced and Correlation is increased.

To check the main hypothesis, we have performed a statistical comparison between all approaches in terms of their ROS Sharpe ratio. Table 3 present the results of paired t-tests, controlling for multiple hypothesis testing using Bonferroni correction. Overall, three main patterns can be spotted:

• •

All Covariance-Penalty approaches outperformed significantly the Naive approach; and our proposed two, Imp SR and R-Squared, also compares favourably in relation to the AIC.

• •

R-Squared provided slightly better results than Imp SR, but its effect-size is small when compared to AIC and Naive cases.

• •

TLS improves ROS Sharpe ratio in relation to OLS; as we have outlined before, since Sharpe ratio and Correlation are strongly associated, as TLS is maximizing Correlation, it will tend to perform better than OLS in relation to this metric.

There is also another explanation of why TLS has outperformed OLS: Figure 9 outline the similarity between approaches in terms of ROS Sharpe ration, and the most frequent (mode) lag selected across the in-out sample sets, averaged across 1361 assets. Overall, TLS penalized harder than OLS, making it many setups to adopt smaller lags, which in some sense, mimic a mean-reversion strategies; similarly, whenever OLS preferred smaller Hypothesis Spaces (less parameters), ROS Sharpe ratio rose. Overall ROS Sharpe ratio are quite correlated across the different methods and types, but some diversification might still be attained by pooling TLS and OLS models.

5 Conclusions

In this paper we proposed a new approach to dealing with financial overfitting, a Covariance-Penalty Correction, in which a risk metric is lowered given the amount of parameters and data used to underpins a trading strategy. We outlined the foundation and main results behind the Covariance-Penalty correction for trading strategies. After that, we pursued an empirical investigation, comparing its performance with some other approaches in the realm of Covariance-Penalties across more than 1300 assets, using Ordinary and Total Least Squares – with the last providing superior performance. Before delving into this novel perspective, we first briefly exhibit a historical context about the available methodologies.

We outlined, chronologically, three distinct approaches in the literature to evaluate and deal with backtesting overfitting: Data Snooping, Overestimated Performance, and Cross-Validation Evaluation. The Data Snooping and Overestimated Performance authors identify that the root cause of any overfitting or spurious results is due to the multiple trials (hypothesis) employed by the analyst during the strategies fine-tuning step. In contrast, the Cross-Validation Evaluation and Covariance-Penalty Correction are focused on avoiding the apparent in-sample performance, by uncovering the expected generalisation (out-sample) potential of a strategy. The Covariance-Penalty Correction methods require some assumptions on the joint behaviour of the asset and trading strategy, which is not the case for the majority of the Cross-Validation procedures. However, this lack of assumptions demand an extra computational processing in respect to the Covariance-Penalty approach.

We then focused on to address the issue of overfitting of dynamic models, we look to estimated objective functions, taking degrees of freedom into account. Most commonly used estimates include Akaike and Bayes’ information criteria, but these require the use of a likelihood. In our case, we do not have an exact likelihood, but instead choose to optimise the Sharpe ratio, and noting the monotonicity in $\rho$, this comes from maximizing $\hat{\rho}$. Consequently, we wish to consider the means of optimising correlation taking into account penalisations for degrees of freedom. In this sense, we presented formulas for adjusting estimated in-sample goodness of fits measures, to produce better estimates of the out-of-sample Sharpe ratio performance. Effectively, we would like to have a more sober estimate of the potential performance of our optimised strategies out of sample.

We then performed an empirical investigation, in which the quantitative strategist is required to build a trading strategy based on lagged information of a certain asset returns. The most naive way (called Naive method in our experiments) is to try different lags and to pick the one that maximizes the Sharpe ratio. We compared it with the Akaike Information Criteria (AIC), as well as with the Covariance-Penalty methods proposed in this work: the Implied Sharpe ratio (Imp SR) and the R-Squared methods. In summary, our main hypothesis is: Can the proposed approaches, Imp SR and R-Squared,outperform AIC and Naive in terms of out-sample/realised Sharpe ratio? In addition to this, we verified the level of alignment between the Expected/In-Sample and Realised/Out-sample Sharpe ratios. With that, our goal was to analyse if there was any additional benefit in terms of performance estimation that can be made by following a Covariance-Penalty approach.

Our findings suggest that Realised Out-Sample (ROS) Sharpe ratio, using some Covariance-Penalty method provides a boost in the unconditional average of 80-100% percent in relation to the Naive approach. Our two proposed approaches, Imp SR and R-squared, also improves 20-30% in relation to the AIC lag selection. Overall, this pattern is also visible in terms of alignment between Expected and Realised Sharpe ratios: MAD is reduced and Correlation is increased. Overall, all Covariance-Penalty approaches outperformed significantly the Naive approach; and our proposed two, Imp SR and R-Squared, also compares favourably in relation to the AIC. Total Least Squares (TLS) improves ROS Sharpe ratio in relation to Ordinary Least Squares (OLS). One of the explanations why TLS outcompeted OLS was that TLS penalized harder than OLS, making it many setups to adopt smaller lags, which in some sense, mimic a mean-reversion strategies. Similarly, whenever OLS preferred smaller Hypothesis Spaces (less parameters), ROS Sharpe ratio rose. Overall ROS Sharpe ratio are quite correlated across the different methods and types, but some diversification might still be attained by pooling TLS and OLS models.

Acknowledgement

Adriano Koshiyama would like to acknowledge the National Research Council of Brazil for his PhD scholarship.

Appendix A Full distributions for single period

In general, for $X$ and $R$ jointly normal, we have $X_{t}R_{t}$ is known to have a pdf,

[TABLE]

where $K_{0}(\cdot)$ is a modified Bessel function of the $2^{nd}$ kind ([43], p 51, eq 6.15). The more general density for non-zero means, is given in [9] as an infinite series. In the special cases of independence and of correlated but zero mean, the expressions become much simpler and we choose to focus on the zero-mean case here. The density is unbounded at zero and has fat tails and positive skewness, becoming more pronounced with higher correlation. We can see the distribution for a variety of correlations in figure (10), with the skewness becoming increasingly pronounced for higher $\rho$. In the limit as $\rho\rightarrow 1$ the distribution converges to that of the central $\chi^{2}$ distribution with one degree of freedom.

Appendix B Nonzero means: Sharpe ratios and Skewness

By an abuse of notation, we define $SR[R]$ to be $\mu_{R}/\sigma_{R}$ and by an abuse of notation, we define $SR[X]=\mu_{X}/\sigma_{X}$ (for $X$ the signal),

Corollary 1: If $R\sim\mathscr{N}(\mu_{R},\sigma_{R}^{2})$ and $X\sim\mathscr{N}(\mu_{X},\sigma_{X}^{2})$ then

[TABLE]

Corollary 2: If $R\sim\mathscr{N}(\mu_{R},\sigma_{R}^{2})$ and $X\sim\mathscr{N}(\mu_{X},\sigma_{X}^{2})$ then

[TABLE]

We note the one period Sharpe ratio of the strategy may depend on both the interaction between the Sharpe ratios of the Signals (weights) and the Returns, in particular whether they have the same sign or not, together with the sign of the correlation. In fact, the amplitude of the resulting strategy SR may be more dependent on the respective Sharpe ratios rather than $\rho$ since after all, $-1\leq\rho\leq 1,$while $SR[R]$ and $SR[X]$ may individually be above $1$.