Asset Pricing in Pre-trained Transformer

TL;DR

This paper introduces the SERT Transformer model for US stock pricing, demonstrating its superior performance during market shocks and its ability to capture temporal sparsity in volatile asset data.

Contribution

The paper develops the SERT model and applies pre-trained Transformers to asset pricing, showing improved accuracy and risk hedging during extreme market conditions.

Findings

SERT outperforms standard Transformers in out-of-sample R2 during COVID-19.

Transformer models enhance risk hedging, with higher Sortino ratios in pandemic periods.

Pre-trained Transformers effectively capture temporal sparsity in volatile asset data.

Abstract

This paper proposes an innovative Transformer model, Single-directional representative from Transformer (SERT), for US large capital stock pricing. It also innovatively applies the pre-trained Transformer models under the stock pricing and factor investment context. They are compared with standard Transformer models and encoder-only Transformer models in three periods covering the entire COVID-19 pandemic to examine the model adaptivity and suitability during the extreme market fluctuations. Namely, pre-COVID-19 period (mild up-trend), COVID-19 period (sharp up-trend with deep down shock) and 1-year post-COVID-19 (high fluctuation sideways movement). The best proposed SERT model achieves the highest out-of-sample R2, 11.2% and 10.91% respectively, when extreme market fluctuation takes place followed by pre-trained Transformer models (10.38% and 9.15%). Their Trend-following-based strategy wise performance also proves their excellent capability for hedging downside risks during market shocks. The proposed SERT model achieves a Sortino ratio 47% higher than the buy-and-hold benchmark in the equal-weighted portfolio and 28% higher in the value-weighted portfolio when the pandemic period is attended. It proves that Transformer models have a great capability to capture patterns of temporal sparsity data in the asset pricing factor model, especially with considerable volatilities. We also find the softmax signal filter as the common configuration of Transformer models in alternative contexts, which only eliminates differences between models, but does not improve strategy-wise performance, while increasing attention heads improve the model performance insignificantly and applying the 'layer norm first' method do not boost the model performance in our case.