Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions

TL;DR

This paper introduces a synthetic benchmarking framework to systematically evaluate and compare sequence modeling architectures based on their ability to capture various temporal dependencies using controllable memory functions.

Contribution

It proposes a novel synthetic benchmarking method that generates tasks with specific memory properties to analyze sequence models' strengths and limitations.

Findings

Confirmed existing theoretical insights into sequence models.

Revealed new behaviors of models with respect to temporal structures.

Highlighted the importance of controllable targets for evaluation.

Abstract

The evolution of sequence modeling architectures, from recurrent neural networks and convolutional models to Transformers and structured state-space models, reflects ongoing efforts to address the diverse temporal dependencies inherent in sequential data. Despite this progress, systematically characterizing the strengths and limitations of these architectures remains a fundamental challenge. In this work, we propose a synthetic benchmarking framework to evaluate how effectively different sequence models capture distinct temporal structures. The core of this approach is to generate synthetic targets, each characterized by a memory function and a parameter that determines the strength of temporal dependence. This setup allows us to produce a continuum of tasks that vary in temporal complexity, enabling fine-grained analysis of model behavior concerning specific memory properties. We focus on four representative memory functions, each corresponding to a distinct class of temporal structures. Experiments on several sequence modeling architectures confirm existing theoretical insights and reveal new findings. These results demonstrate the effectiveness of the proposed method in advancing theoretical understanding and highlight the importance of using controllable targets with clearly defined structures for evaluating sequence modeling architectures.