CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation

TL;DR

This paper introduces CreativeEval, a framework for quantifying the creativity of large language models in hardware code generation, focusing on aspects like originality and flexibility, and evaluates several models including GPT-3.5.

Contribution

The paper proposes a novel framework for measuring creativity in LLM-generated hardware designs, addressing a gap in existing evaluation methods.

Findings

GPT-3.5 exhibits the highest creativity among tested models.

CreativeEval effectively quantifies creativity in hardware code generation.

The framework distinguishes models based on fluency, flexibility, originality, and elaboration.

Abstract

Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs.