Hybrid-NL2SVA: Integrating RAG and Finetuning for LLM-based NL2SVA

TL;DR

This paper introduces Hybrid-NL2SVA, a novel approach combining retrieval-augmented generation and fine-tuning to improve the automatic translation of natural language descriptions into SystemVerilog Assertions for hardware verification.

Contribution

It presents a customized RAG framework and a synthetic fine-tuning dataset that significantly enhance LLM performance in NL2SVA tasks, especially for domain-specific syntax and semantics.

Findings

RAG framework increases functionality matched SVAs by 58.42%.

Fine-tuned models achieve 59.05% improvement over base models.

Largest NL2SVA evaluation dataset with 40 designs and 229 SVAs.

Abstract

SystemVerilog Assertions (SVAs) are critical for verifying the correctness of hardware designs, but manually writing them from natural language property descriptions, i.e., NL2SVA, remains a labor-intensive and error-prone task. Recent advances in large language models (LLMs) offer opportunities to automate this translation. However, existing models still struggle with understanding domain-specific syntax and semantics. To enhance LLM performance in NL2SVA, we propose a customized retrieval-augmented generation (RAG) framework and a synthetic fine-tuning dataset that together improve LLM's performance. To further improve lightweight models over NL2SVA, our fine-tuning dataset provides prompt-guided explanations that teach LLMs the layer-by-layer construction process of concurrent SVAs, enabling supervised fine-tuning that greatly improves syntax and functionality accuracy. To evaluate the performance of LLMs over NL2SVA, we construct the largest evaluation dataset for NL2SVA, comprising 40 Verilog designs and 229 formally verified SVAs with detailed annotations. Experimental results show that our customized RAG framework increases the number of functionality matched SVAs by 58.42% over GPT-4o-mini, while Qwen2.5-Coder-7B-Instruct fine-tuned on our fine-tuning dataset and integrated with HybridRetrieval achieves a 59.05% over the base Qwen model.