Implementation Risk in Portfolio Backtesting: A Previously Unquantified Source of Error

TL;DR

This paper formalizes implementation risk in portfolio backtesting, quantifies it with new metrics, and reveals its impact on performance divergence across different engines and transaction costs.

Contribution

It introduces four metrology-based metrics to quantify implementation risk and uncovers undocumented defects in backtesting engines.

Findings

Engine divergence increases with transaction costs.

Most strategies show divergence below 0.75%, high-turnover strategies up to 3.71%.

Seven previously undocumented engine defects were identified.

Abstract

Portfolio backtesting is the primary tool for evaluating investment strategies before deployment, yet practitioners implicitly assume that different engines produce identical results for the same strategy. we formalise implementation risk, the systematic divergence in backtested portfolio metrics arising solely from differences in how engines implement the same logical strategy, and propose four metrics grounded in metrology to quantify it: engine sensitivity, implementation uncertainty interval, divergence amplification factor, and conclusion stability index. we execute 15 benchmark strategies through five independent open-source engines on 30 non-overlapping stratified asset buckets comprising 180 s&p 500 stocks under four transaction-cost regimes. at zero cost, all five engines agree exactly (maximum divergence 0.000%), isolating transaction-cost implementation as the sole source of disagreement. under nonzero costs, divergence is structured and predictable (spearman rho = 0.93 with cost intensity), remaining below 0.75 percentage points for most strategies but reaching 3.71% for high-turnover rotation strategies. source-code forensics uncovered seven previously undocumented defects across three engines, abstracted into a five-category failure-mode taxonomy. all engines agree on the sign of every performance metric (conclusion stability index = 1), so implementation risk does not alter investment decisions for the strategies studied but introduces measurable ambiguity in performance attribution. code and benchmark data are publicly available.