FACT: Learning Governing Abstractions Behind Integer Sequences

TL;DR

This paper introduces FACT, a toolkit and benchmarking suite for evaluating machine learning models' ability to learn and reason about integer sequences, emphasizing abstraction, interpolation, and extrapolation.

Contribution

The paper presents a novel benchmarking framework and toolkit for assessing machine learning models' understanding of integer sequences and their ability to abstract concepts.

Findings

Benchmark tasks for conceptual understanding of integer sequences

A large dataset of organic and synthetic sequences

Baseline models for evaluation

Abstract

Integer sequences are of central importance to the modeling of concepts admitting complete finitary descriptions. We introduce a novel view on the learning of such concepts and lay down a set of benchmarking tasks aimed at conceptual understanding by machine learning models. These tasks indirectly assess model ability to abstract, and challenge them to reason both interpolatively and extrapolatively from the knowledge gained by observing representative examples. To further aid research in knowledge representation and reasoning, we present FACT, the Finitary Abstraction Comprehension Toolkit. The toolkit surrounds a large dataset of integer sequences comprising both organic and synthetic entries, a library for data pre-processing and generation, a set of model performance evaluation tools, and a collection of baseline model implementations, enabling the making of the future advancements with ease.