Lattice Deduction Transformers

TL;DR

The paper presents the Lattice Deduction Transformer (LDT), a recurrent transformer model that approximates logical deduction, achieving high accuracy on Sudoku and Maze benchmarks with efficient training.

Contribution

It introduces a novel recurrent transformer architecture that performs logical deduction via lattice projection, with domain-agnostic supervision and strong empirical results.

Findings

LDT achieves 100% accuracy on Sudoku-Extreme and Snowflake Sudoku.

A larger LDT variant reaches 99.9% accuracy on Maze-Hard.

Frontier LLMs score 0% on all benchmarks.

Abstract

We introduce the Lattice Deduction Transformer (LDT), a recurrent transformer that approximates logically sound deduction by projecting its latent state through a lattice between forward passes. We train on-policy in a process that mirrors deduction in a search-based constraint solver and supervise training via a domain-agnostic, abstract-interpretation-based approximation of the set of solution candidates. An $800$K-parameter LDT achieves $100\%$ accuracy on Sudoku-Extreme and Snowflake Sudoku, at a fraction of the training cost of prior small recurrent reasoners, while remaining empirically sound: the model returns a correct answer or abstains. A $1.8$M-parameter variant reaches $99.9\%$ accuracy on Maze-Hard. Frontier LLMs score $0\%$ on all three benchmarks.