FaCTR: Factorized Channel-Temporal Representation Transformers for Efficient Time Series Forecasting

TL;DR

FaCTR is a lightweight, structurally designed Transformer model that efficiently captures complex multivariate dependencies in time series data, achieving state-of-the-art forecasting performance with significantly fewer parameters.

Contribution

FaCTR introduces a novel factorized, structural Transformer architecture tailored for time series forecasting, emphasizing efficiency, interpretability, and versatility.

Findings

Achieves state-of-the-art results on 11 benchmarks.

Uses 50x fewer parameters than competitors.

Supports self-supervised pretraining.

Abstract

While Transformers excel in language and vision-where inputs are semantically rich and exhibit univariate dependency structures-their architectural complexity leads to diminishing returns in time series forecasting. Time series data is characterized by low per-timestep information density and complex dependencies across channels and covariates, requiring conditioning on structured variable interactions. To address this mismatch and overparameterization, we propose FaCTR, a lightweight spatiotemporal Transformer with an explicitly structural design. FaCTR injects dynamic, symmetric cross-channel interactions-modeled via a low-rank Factorization Machine into temporally contextualized patch embeddings through a learnable gating mechanism. It further encodes static and dynamic covariates for multivariate conditioning. Despite its compact design, FaCTR achieves state-of-the-art performance on eleven public forecasting benchmarks spanning both short-term and long-term horizons, with its largest variant using close to only 400K parameters-on average 50x smaller than competitive spatiotemporal transformer baselines. In addition, its structured design enables interpretability through cross-channel influence scores-an essential requirement for real-world decision-making. Finally, FaCTR supports self-supervised pretraining, positioning it as a compact yet versatile foundation for downstream time series tasks.