HWE-Bench: Can Language Models Perform Board-level Schematic Designs?

TL;DR

HWE-Bench evaluates how well large language models can perform complex board-level schematic designs by testing their ability to generate functional circuits based on detailed specifications and datasheets.

Contribution

This paper introduces HWE-Bench, a comprehensive benchmark with 300 tasks and a datasheet knowledge base to assess LLMs' capabilities in board-level circuit design.

Findings

Current models achieve only 8.15% pass rate.

Models understand documentation but lack physical intuition.

Benchmark highlights gaps in LLMs' circuit design skills.

Abstract

Large Language Models (LLMs) have demonstrated significant potential in various engineering tasks, including software development, digital logic generation, and companion document maintenance. However, their ability to perform board-level circuit design is understudied, as this task requires a synergized understanding of real-world physics and Integrated Circuit (IC) datasheets, the latter comprising detailed specifications for individual components. To address this challenge, we propose \hweb, an evaluation framework that benchmarks the ability of LLMs to perform such designs. It consists of 300 board-level design tasks pulled from open-source and crowdsourcing platforms such as GitHub and OSHWLab, covering 8 application domains, and is complemented with a knowledge base of 2,914 real IC datasheets. For each task, the LLMs are tasked with generating a schematic from scratch, using the provided circuit functional requirements and a set of component datasheets as input. The resulting schematic will be checked against a static electrical rules, and then passed to a circuit simulator to verify its dynamic behavior. Our evaluation show that although current models achieve initial engineering usability and documentation understanding, they lack physical intuition, as the top-performing model achieved an overall pass rate of 8.15\%. We envision that advancements on \hweb\ will pave the way for the development of practical Electronic Design Automation (EDA) agents, revolutionizing the field of board-level design.