ArchXBench: A Complex Digital Systems Benchmark Suite for LLM Driven RTL Synthesis

TL;DR

ArchXBench is a comprehensive benchmark suite for evaluating LLMs' ability to generate complex RTL digital systems, revealing current limitations of state-of-the-art models in handling advanced design tasks.

Contribution

The paper introduces ArchXBench, a new multi-level benchmark suite with complex digital circuits, and evaluates LLMs' performance, highlighting gaps in their capabilities for RTL synthesis.

Findings

o4-mini-high solves 16 out of 30 benchmarks

Models fail on levels 4 and above

Current LLMs have limited ability for complex RTL design

Abstract

Modern SoC datapaths include deeply pipelined, domain-specific accelerators, but their RTL implementation and verification are still mostly done by hand. While large language models (LLMs) exhibit advanced code-generation abilities for programming languages like Python, their application to Verilog-like RTL remains in its nascent stage. This is reflected in the simple arithmetic and control circuits currently used to evaluate generative capabilities in existing benchmarks. In this paper, we introduce ArchXBench, a six-level benchmark suite that encompasses complex arithmetic circuits and other advanced digital subsystems drawn from domains such as cryptography, image processing, machine learning, and signal processing. Architecturally, some of these designs are purely combinational, others are multi-cycle or pipelined, and many require hierarchical composition of modules. For each benchmark, we provide a problem description, design specification, and testbench, enabling rapid research in the area of LLM-driven agentic approaches for complex digital systems design. Using zero-shot prompting with Claude Sonnet 4, GPT 4.1, o4-mini-high, and DeepSeek R1 under a pass@5 criterion, we observed that o4-mini-high successfully solves the largest number of benchmarks, 16 out of 30, spanning Levels 1, 2, and 3. From Level 4 onward, however, all models consistently fail, highlighting a clear gap in the capabilities of current state-of-the-art LLMs and prompting/agentic approaches.