ChiBench: a Benchmark Suite for Testing Electronic Design Automation Tools

TL;DR

ChiBench is a large, curated benchmark suite of 50,000 Verilog programs from GitHub, designed to test and evaluate electronic design automation tools' robustness, complexity handling, and coverage.

Contribution

This paper introduces ChiBench, the first extensive benchmark suite with 50,000 Verilog programs, enabling comprehensive testing of EDA tools.

Findings

Revealed bugs in public EDA tools like Verible.

Demonstrated ChiBench’s use in evaluating tool complexity and coverage.

Validated the effectiveness of ChiBench in identifying tool issues.

Abstract

Electronic Design Automation (EDA) tools are software applications used by engineers in the design, development, simulation, and verification of electronic systems and integrated circuits. These tools typically process specifications written in a Hardware Description Language (HDL), such as Verilog, SystemVerilog or VHDL. Thus, effective testing of these tools requires benchmark suites written in these languages. However, while there exist some open benchmark suites for these languages, they tend to consist of only a handful of specifications. This paper, in contrast, presents ChiBench, a comprehensive suite comprising 50 thousand Verilog programs. These programs were sourced from GitHub repositories and curated using Verible's syntactic analyzer and Jasper(TM)'s HDL semantic analyzer. Since its inception, ChiBench has already revealed bugs in public tools like Verible's obfuscator and parser. In addition to explaining some of these case studies, this paper demonstrates how ChiBench can be used to evaluate the asymptotic complexity and code coverage of typical electronic design automation tools.