CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs

TL;DR

This paper introduces CIRCUIT, a benchmark dataset to evaluate LLMs' reasoning capabilities in analog circuit interpretation, revealing current models' limitations in understanding complex circuit topologies and reasoning tasks.

Contribution

The paper presents the first dedicated benchmark for assessing LLMs' reasoning in analog circuit design, including a novel unit-test evaluation method.

Findings

GPT-4o achieves 48.04% accuracy on final answers.

GPT-4o passes only 27.45% of unit tests.

Current LLMs struggle with multi-level circuit reasoning.

Abstract

The role of Large Language Models (LLMs) has not been extensively explored in analog circuit design, which could benefit from a reasoning-based approach that transcends traditional optimization techniques. In particular, despite their growing relevance, there are no benchmarks to assess LLMs' reasoning capability about circuits. Therefore, we created the CIRCUIT dataset consisting of 510 question-answer pairs spanning various levels of analog-circuit-related subjects. The best-performing model on our dataset, GPT-4o, achieves 48.04% accuracy when evaluated on the final numerical answer. To evaluate the robustness of LLMs on our dataset, we introduced a unique feature that enables unit-test-like evaluation by grouping questions into unit tests. In this case, GPT-4o can only pass 27.45% of the unit tests, highlighting that the most advanced LLMs still struggle with understanding circuits, which requires multi-level reasoning, particularly when involving circuit topologies. This circuit-specific benchmark highlights LLMs' limitations, offering valuable insights for advancing their application in analog integrated circuit design.