Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation

TL;DR

Bench4HLS is a comprehensive benchmarking framework for evaluating the quality and effectiveness of large language models in generating high-level synthesis code for hardware design, supporting automated assessments and PPA analysis.

Contribution

This paper introduces Bench4HLS, the first extensive, automated benchmarking framework for LLM-generated HLS code, including a large curated dataset and PPA analysis capabilities.

Findings

Bench4HLS successfully evaluates LLM-generated HLS designs across multiple metrics.

The framework supports integration with different HLS tools and architectures.

Bench4HLS provides insights into the strengths and limitations of LLMs in hardware synthesis.

Abstract

In last two years, large language models (LLMs) have shown strong capabilities in code generation, including hardware design at register-transfer level (RTL). While their use in high-level synthesis (HLS) remains comparatively less mature, the ratio of HLS- to RTL-focused studies has shifted from 1:10 to 2:10 in the past six months, indicating growing interest in leveraging LLMs for high-level design entry while relying on downstream synthesis for optimization. This growing trend highlights the need for a comprehensive benchmarking and evaluation framework dedicated to LLM-based HLS. To address this, We present Bench4HLS for evaluating LLM-generated HLS designs. Bench4HLS comprises 170 manually drafted and validated case studies, spanning small kernels to complex accelerators, curated from widely used public repositories. The framework supports fully automated assessment of compilation success, functional correctness via simulation, and synthesis feasibility/optimization. Crucially, Bench4HLS integrates a pluggable API for power, performance, and area (PPA) analysis across various HLS toolchains and architectures, demonstrated here with Xilinx Vitis HLS and validated on Catapult HLS. By providing a structured, extensible, and plug-and-play testbed, Bench4HLS establishes a foundational methodology for benchmarking LLMs in HLS workflows.