Early Discoveries of Algorithmist I: Promise of Provable Algorithm Synthesis at Scale

TL;DR

This paper presents Algorithmist, an autonomous agent leveraging LLMs to synthesize provably sound algorithms tailored to specific datasets, demonstrating practical effectiveness and uncovering new insights in algorithm design and verification.

Contribution

It introduces a novel LLM-based research agent capable of on-the-fly provable algorithm synthesis, bridging theory and practice without prior distributional assumptions.

Findings

Produced provably sound algorithms for data privacy and clustering tasks

Uncovered a subtle proof bug in prior work

Generated research-quality algorithmic artifacts tailored to datasets

Abstract

Designing algorithms with provable guarantees that also work well in practice remains difficult, requiring both mathematical reasoning and careful implementation. Existing approaches that bridge worst-case theory and empirical performance, such as beyond-worst-case analysis and data-driven algorithm selection, typically assume prior distributional knowledge or restrict attention to a fixed pool of algorithms. Recent progress in LLMs suggests a new possibility: provable algorithm synthesis on the fly. To study this, we built Algorithmist, an autonomous researcher agent on top of GitHub Copilot that runs a multi-agent research-and-review loop, with separate stages for idea generation, algorithm and proof development, proof-guided implementation, and review of proofs, code, and their alignment. We evaluate Algorithmist on research-level tasks in private data analysis and clustering. When asked to design practical methods that jointly satisfy privacy, approximation, and interpretability requirements, it produced provably sound and empirically effective algorithms, together with research-style writeups and audited implementations. It also found improved algorithms in some settings, explained principled barriers in others, and uncovered a subtle proof bug in prior published work. More broadly, our results suggest a new paradigm in which LLM systems generate research-paper-quality algorithmic artifacts tailored to each dataset and deployment setting. They also point to a proof-first code-synthesis paradigm, in which code is developed alongside a structured natural-language proof intermediate representation and kept aligned with it throughout synthesis.