Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes

TL;DR

This paper evaluates 32 language models on RTL generation tasks using a new Hardware Quality Index, revealing performance regimes, failure modes, and the impact of training data on synthesis quality.

Contribution

It introduces the Hardware Quality Index for comprehensive RTL synthesis evaluation and provides a detailed failure taxonomy across models.

Findings

Frontier models achieve HQI > 66 with Gemini-3-Pro at 87.5% coverage.

Systematic divergence in failure modes between proprietary and open models.

Training data skewed toward simulation affects synthesis success.

Abstract

RTL generation is more than code synthesis. Designs must be syntactically valid, synthesizable, correct, hardware-efficient. SOTA evaluations stop at functional correctness and do not measure synthesis and implementation quality. This paper evaluates 32 language models on 202 Verilog tasks from VerilogEval and RTLLM using the Hardware Quality Index (HQI) that combines post-synthesis area, delay, and warnings related to expert references in a Nangate45 45\,nm flow. Three performance regimes emerge: 14 frontier models achieve HQI $>$ 66, led by Gemini-3-Pro at 87.5\% coverage and 85.1 HQI; 15 models cluster 43--66 HQI; 3 are below 43. Gap between best-of-five capability and single-attempt quality spans 3.7--22.1 HQI points, limiting integration into agentic pipelines. A taxonomy of 195 synthesis failures reveals systematic divergence: proprietary models fail late through elaboration errors and synthesis timeout; open models fail early often due to missing module wrappers and non-synthesizable constructs, a pattern consistent with training corpora skewed toward simulation over synthesis-grade RTL.