MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark

TL;DR

MHRC-Bench is a pioneering multilingual hardware repository-level code completion benchmark that evaluates models on hardware description languages, covering multiple coding styles and providing detailed semantic annotations.

Contribution

This work introduces the first benchmark for multilingual hardware code completion at the repository level, including annotated datasets and comprehensive evaluation protocols.

Findings

Models show varying performance across hardware languages

Semantic annotations improve understanding of hardware code structure

Benchmark effectively differentiates model capabilities in hardware code completion

Abstract

Large language models (LLMs) have achieved strong performance on code completion tasks in general-purpose programming languages. However, existing repository-level code completion benchmarks focus almost exclusively on software code and largely overlook hardware description languages. In this work, we present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark designed for multilingual hardware code completion at the repository level. Our benchmark targets completion tasks and covers three major hardware design coding styles. Each completion target is annotated with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree analysis. We conduct a comprehensive evaluation of models on MHRC-Bench-Eval. Comprehensive evaluation results and analysis demonstrate the effectiveness of MHRC-Bench.