SymSeqBench: a unified framework for the generation and analysis of rule-based symbolic sequences and datasets

TL;DR

SymSeqBench is a versatile, formal language theory-based framework comprising tools for generating, analyzing, and benchmarking rule-based symbolic sequences across diverse cognitive and AI domains.

Contribution

It introduces SymSeq and SeqBench as a unified, formal language theory-based platform for sequence generation, analysis, and benchmarking in cognition and AI research.

Findings

Provides a domain-agnostic framework for sequence analysis

Enables standardized evaluation of AI systems on rule-based tasks

Links cognitive modeling with formal language theory

Abstract

Sequential structure is a key feature of multiple domains of natural cognition and behavior, such as language, movement and decision-making. Likewise, it is also a central property of tasks to which we would like to apply artificial intelligence. It is therefore of great importance to develop frameworks that allow us to evaluate sequence learning and processing in a domain agnostic fashion, whilst simultaneously providing a link to formal theories of computation and computability. To address this need, we introduce two complementary software tools: SymSeq, designed to rigorously generate and analyze structured symbolic sequences, and SeqBench, a comprehensive benchmark suite of rule-based sequence processing tasks to evaluate the performance of artificial learning systems in cognitively relevant domains. In combination, SymSeqBench offers versatility in investigating sequential structure across diverse knowledge domains, including experimental psycholinguistics, cognitive psychology, behavioral analysis, neuromorphic computing and artificial intelligence. Due to its basis in Formal Language Theory (FLT), SymSeqBench provides researchers in multiple domains with a convenient and practical way to apply the concepts of FLT to conceptualize and standardize their experiments, thus advancing our understanding of cognition and behavior through shared computational frameworks and formalisms. The tool is modular, openly available and accessible to the research community.