AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design

TL;DR

AutoBench leverages large language models to automatically generate and evaluate comprehensive testbenches for digital circuit design, significantly improving efficiency and success rates over direct LLM-based generation methods.

Contribution

AutoBench introduces the first LLM-based automated testbench generator with a hybrid structure and evaluation framework for digital circuits.

Findings

57% improvement in testbench pass@1 ratio over baseline

Successfully applied to 75 sequential circuits with 3.36 times higher pass@1 ratio

Open-sourced implementation and experimental results available

Abstract

In digital circuit design, testbenches constitute the cornerstone of simulation-based hardware verification. Traditional methodologies for testbench generation during simulation-based hardware verification still remain partially manual, resulting in inefficiencies in testing various scenarios and requiring expensive time from designers. Large Language Models (LLMs) have demonstrated their potential in automating the circuit design flow. However, directly applying LLMs to generate testbenches suffers from a low pass rate. To address this challenge, we introduce AutoBench, the first LLM-based testbench generator for digital circuit design, which requires only the description of the design under test (DUT) to automatically generate comprehensive testbenches. In AutoBench, a hybrid testbench structure and a self-checking system are realized using LLMs. To validate the generated testbenches, we also introduce an automated testbench evaluation framework to evaluate the quality of generated testbenches from multiple perspectives. Experimental results demonstrate that AutoBench achieves a 57% improvement in the testbench pass@1 ratio compared with the baseline that directly generates testbenches using LLMs. For 75 sequential circuits, AutoBench successfully has a 3.36 times testbench pass@1 ratio compared with the baseline. The source codes and experimental results are open-sourced at this link: https://github.com/AutoBench/AutoBench