PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models

TL;DR

PCEval is a novel benchmark that automatically evaluates large language models' abilities in physical computing tasks, including circuit design and code generation, revealing strengths in logical reasoning but challenges in physical layout creation.

Contribution

Introduces PCEval, the first fully automatic benchmark for assessing LLMs' physical computing capabilities in both logical and physical aspects.

Findings

LLMs excel in code generation and logical circuit design.

LLMs struggle with physical breadboard layout and pin connection management.

PCEval provides a reproducible framework for evaluating LLMs in hardware-related tasks.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, including software development, education, and technical assistance. Among these, software development is one of the key areas where LLMs are increasingly adopted. However, when hardware constraints are considered-for instance, in physical computing, where software must interact with and control physical hardware -their effectiveness has not been fully explored. To address this gap, we introduce \textsc{PCEval} (Physical Computing Evaluation), the first benchmark in physical computing that enables a fully automatic evaluation of the capabilities of LLM in both the logical and physical aspects of the projects, without requiring human assessment. Our evaluation framework assesses LLMs in generating circuits and producing compatible code across varying levels of project complexity. Through comprehensive testing of 13 leading models, \textsc{PCEval} provides the first reproducible and automatically validated empirical assessment of LLMs' ability to reason about fundamental hardware implementation constraints within a simulation environment. Our findings reveal that while LLMs perform well in code generation and logical circuit design, they struggle significantly with physical breadboard layout creation, particularly in managing proper pin connections and avoiding circuit errors. \textsc{PCEval} advances our understanding of AI assistance in hardware-dependent computing environments and establishes a foundation for developing more effective tools to support physical computing education.