A rigorous version of R. P. Brent's model for the binary Euclidean algorithm

TL;DR

This paper rigorously analyzes Brent's heuristic model for the binary Euclidean algorithm, proving key properties of the associated transfer operator and confirming conjectured formulas for the average number of steps.

Contribution

It establishes spectral gap and density properties of Brent's transfer operator, connecting heuristic models with rigorous proofs and resolving open questions.

Findings

Proves the transfer operator has a spectral gap.

Shows the density extends holomorphically with a logarithmic singularity.

Confirms conjectured formulas for the expected number of steps.

Abstract

The binary Euclidean algorithm is a modification of the classical Euclidean algorithm for computation of greatest common divisors which avoids ordinary integer division in favour of division by powers of two only. The expectation of the number of steps taken by the binary Euclidean algorithm when applied to pairs of integers of bounded size was first investigated by R. P. Brent in 1976 via a heuristic model of the algorithm as a random dynamical system. Based on numerical investigations of the expectation of the associated Ruelle transfer operator, Brent obtained a conjectural asymptotic expression for the mean number of steps performed by the algorithm when processing pairs of odd integers whose size is bounded by a large integer. In 1998 B. Vall\'ee modified Brent's model via an induction scheme to rigorously prove an asymptotic formula for the average number of steps performed by the algorithm; however, the relationship of this result with Brent's heuristics remains conjectural. In this article we establish previously conjectural properties of Brent's transfer operator, showing directly that it possesses a spectral gap and preserves a unique continuous density. This density is shown to extend holomorphically to the complex right half-plane and to have a logarithmic singularity at zero. By combining these results with methods from classical analytic number theory we prove the correctness of three conjectured formulae for the expected number of steps, resolving several open questions promoted by D. E. Knuth in The Art of Computer Programming.