Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization

TL;DR

This paper evaluates the performance of existing code LLMs on VHDL code generation and summarization, finds significant underperformance, and introduces Chain-of-Descriptions (CoDes), a novel prompting method that substantially improves results.

Contribution

The paper introduces CoDes, a new prompting strategy that enhances LLMs' ability to generate and summarize VHDL code by incorporating intermediate descriptive steps.

Findings

CoDes significantly outperforms standard prompts in VHDL tasks

Existing LLMs underperform on VHDL code generation and summarization

The approach provides a framework for future HDL model improvements

Abstract

Large Language Models (LLMs) have become widely used across diverse NLP tasks and domains, demonstrating their adaptability and effectiveness. In the realm of Electronic Design Automation (EDA), LLMs show promise for tasks like Register-Transfer Level (RTL) code generation and summarization. However, despite the proliferation of LLMs for general code-related tasks, there's a dearth of research focused on evaluating and refining these models for hardware description languages (HDLs), notably VHDL. In this study, we evaluate the performance of existing code LLMs for VHDL code generation and summarization using various metrics and two datasets -- VHDL-Eval and VHDL-Xform. The latter, an in-house dataset, aims to gauge LLMs' understanding of functionally equivalent code. Our findings reveal consistent underperformance of these models across different metrics, underscoring a significant gap in their suitability for this domain. To address this challenge, we propose Chain-of-Descriptions (CoDes), a novel approach to enhance the performance of LLMs for VHDL code generation and summarization tasks. CoDes involves generating a series of intermediate descriptive steps based on: (i) the problem statement for code generation, and (ii) the VHDL code for summarization. These steps are then integrated with the original input prompt (problem statement or code) and provided as input to the LLMs to generate the final output. Our experiments demonstrate that the CoDes approach significantly surpasses the standard prompting strategy across various metrics on both datasets. This method not only improves the quality of VHDL code generation and summarization but also serves as a framework for future research aimed at enhancing code LLMs for VHDL.