Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation

TL;DR

This paper introduces a multi-modal benchmark and query framework for Verilog generation that combines visual and natural language inputs, showing improved accuracy over language-only methods in hardware design tasks.

Contribution

It presents an open-source multi-modal benchmark and query language framework for Verilog synthesis, highlighting the importance of visual context in hardware design automation.

Findings

Multi-modal models outperform natural language-only models in Verilog accuracy.

Open-source benchmark and query framework facilitate multi-modal hardware design.

Significant accuracy improvements demonstrate the value of visual information in Verilog generation.

Abstract

Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singular and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a significant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large-hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.