Nonlinear Dynamic Factor Analysis With a Transformer Network

TL;DR

This paper introduces a Transformer-based model for dynamic factor analysis in multivariate time series, improving accuracy over linear models, especially with small datasets, and providing interpretability through attention mechanisms.

Contribution

It develops a novel Transformer architecture for dynamic factor estimation, incorporating prior information and interpretability, and demonstrates superior performance over traditional linear models.

Findings

Transformer outperforms linear models in accuracy.

Attention patterns reveal regime switches.

Regularization improves small dataset performance.

Abstract

The paper develops a Transformer architecture for estimating dynamic factors from multivariate time series data under flexible identification assumptions. Performance on small datasets is improved substantially by using a conventional factor model as prior information via a regularization term in the training objective. The results are interpreted with Attention matrices that quantify the relative importance of variables and their lags for the factor estimate. Time variation in Attention patterns can help detect regime switches and evaluate narratives. Monte Carlo experiments suggest that the Transformer is more accurate than the linear factor model, when the data deviate from linear-Gaussian assumptions. An empirical application uses the Transformer to construct a coincident index of U.S. real economic activity.