Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models

TL;DR

Enigme introduces a library of text-based puzzles designed to evaluate and train reasoning abilities in transformer-decoder language models, addressing their architectural limitations in understanding and applying reasoning in natural language tasks.

Contribution

The paper presents Enigme, an open-source tool for generating reasoning puzzles to assess and improve the reasoning capabilities of transformer-based language models.

Findings

Enigme enables targeted evaluation of reasoning in language models.

The puzzles reveal architectural constraints affecting reasoning performance.

Open-source availability facilitates broader research and development.

Abstract

Transformer-decoder language models are a core innovation in text based generative artificial intelligence. These models are being deployed as general-purpose intelligence systems in many applications. Central to their utility is the capacity to understand natural language commands and exploit the reasoning embedded in human text corpora to apply some form of reasoning process to a wide variety of novel tasks. To understand the limitations of this approach to generating reasoning we argue that we need to consider the architectural constraints of these systems. Consideration of the latent variable structure of transformer-decoder models allows us to design reasoning tasks that should probe the boundary of their capacity to reason. We present enigme, an open-source library for generating text-based puzzles to be used in training and evaluating reasoning skills within transformer-decoder models and future AI architectures.