VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generation

TL;DR

This paper empirically analyzes how different language models interact with prompt engineering strategies in Verilog code generation, revealing patterns and general trends across models and benchmarks.

Contribution

It provides an empirical map of recent trends in LM interactions with prompt design in Verilog code generation, including diverse model evaluations and optimization techniques.

Findings

Patterns in model responses to structured prompts and optimization.

Trends that generalize across models and benchmarks.

Identification of model-specific versus general prompt strategies.

Abstract

Rapid advances in language models (LMs) have created new opportunities for automated code generation while complicating trade-offs between model characteristics and prompt design choices. In this work, we provide an empirical map of recent trends in LMs for Verilog code generation, focusing on interactions among model reasoning, specialization, and prompt engineering strategies. We evaluate a diverse set of small and large LMs, including general-purpose, reasoning, and domain-specific variants. Our experiments use a controlled factorial design spanning benchmark prompts, structured outputs, prompt rewriting, chain-of-thought reasoning, in-context learning, and evolutionary prompt optimization via Genetic-Pareto. Across two Verilog benchmarks, we identify patterns in how model classes respond to structured prompts and optimization, and we document which trends generalize across LMs and benchmarks versus those that are specific to particular model-prompt combinations.