FRACTAL: SSM with Fractional Recurrent Architecture for Computational Temporal Analysis of Long Sequences

TL;DR

FRACTAL introduces a novel fractional measure-based architecture for long sequence modeling, balancing global context and local variations more effectively than existing state space models.

Contribution

It integrates fractional measure theory into recursive memory updates, enabling simultaneous multi-scale temporal feature capture with improved performance.

Findings

Achieves 87.11% on Long Range Arena benchmark.

Scores 61.85% on ListOps task, outperforming S5.

Provides spectral analysis of projection operators for scale-invariance.

Abstract

Effective sequence modeling fundamentally requires balancing the retention of unbounded history with the high-resolution detection of abrupt short-term variations common in real-world phenomena. However, existing state space models (SSMs) relying on high-order polynomial projection operators (HiPPO) face a critical trade-off where uniform measures dilute recent information to maintain timescale invariance, while exponential measures sacrifice global context to capture local dynamics. This paper proposes a Fractional Recurrent Architecture for Computational Temporal Analysis of Long sequences (FRACTAL), a novel architecture integrating fractional measure theory into recursive memory updates to address this limitation. By deriving projection operators with analytically characterized spectral properties and a tunable singularity index, the proposed method amplifies sensitivity to recent signal perturbations while preserving the spectral structure that encodes scale-invariant memory dynamics. This theoretical innovation is instantiated within a simplified diagonalized state space framework by modulating input projection initialization to enable simultaneous capture of multi-scale temporal features. FRACTAL achieves an average score of 87.11\% on the Long Range Arena benchmark, including 61.85\% on the ListOps task, outperforming the S5 model.