On the use of LLMs to generate a dataset of Neural Networks

TL;DR

This paper presents a method using large language models to automatically generate a diverse, validated dataset of neural networks for benchmarking and evaluating tools that verify, refactor, and migrate neural architectures.

Contribution

The authors introduce a novel approach leveraging LLMs to create a publicly available, diverse neural network dataset for systematic evaluation of neural network tools.

Findings

Generated 608 neural network samples covering diverse architectures

Validated network correctness using static analysis and symbolic tracing

Dataset supports benchmarking neural network verification and adaptation tools

Abstract

Neural networks are increasingly used to support decision-making. To verify their reliability and adaptability, researchers and practitioners have proposed a variety of tools and methods for tasks such as NN code verification, refactoring, and migration. These tools play a crucial role in guaranteeing both the correctness and maintainability of neural network architectures, helping to prevent implementation errors, simplify model updates, and ensure that complex networks can be reliably extended and reused. Yet, assessing their effectiveness remains challenging due to the lack of publicly diverse datasets of neural networks that would allow systematic evaluation. To address this gap, we leverage large language models (LLMs) to automatically generate a dataset of neural networks that can serve as a benchmark for validation. The dataset is designed to cover diverse architectural components and to handle multiple input data types and tasks. In total, 608 samples are generated, each conforming to a set of precise design choices. To further ensure their consistency, we validate the correctness of the generated networks using static analysis and symbolic tracing. We make the dataset publicly available to support the community in advancing research on neural network reliability and adaptability.