Ehrenfeucht-Haussler Rank and Chain of Thought

TL;DR

This paper introduces a new characterization of Boolean function rank based on Transformer Chain of Thought steps, establishing bounds and implications for PAC learning and multi-head transformer models.

Contribution

It provides a novel Transformer-based rank characterization, tight bounds on CoT steps for specific functions, and analyzes PAC-learnability of functions with bounded multi-head rank.

Findings

Rank corresponds to minimum CoT steps in single-layer Transformers.

Exact CoT steps needed for function composition and position-finding problems.

Analysis of PAC-learnability for functions with bounded multi-head rank.

Abstract

The notion of \emph{rank} of a Boolean function has been a cornerstone in PAC learning theory, enabling quasipolynomial-time learning algorithms for polynomial-size decision trees. We present a novel characterization of rank, grounded in the well-known Transformer architecture. We show that the rank of a function $f$ corresponds to the minimum number of \emph{Chain of Thought} (CoT) steps required by a single-layer Transformer with hard attention to compute $f$. Based on this characterization we establish tight bounds on the number of CoT steps required for specific problems, showing that \(\ell\)-fold function composition necessitates exactly \(\ell\) CoT steps. Furthermore, we analyze the problem of identifying the position of the \(k\)-th occurrence of 1 in a Boolean sequence, proving that it requires \(k\) CoT steps. Finally, we introduce the notion of the multi-head rank that captures multi-head single-layer transformers, and perform the analysis of PAC-learnability of the classes of functions with bounded multi-head rank.