A Novel Mutation Based Method for Detecting FPGA Logic Synthesis Tool Bugs

TL;DR

This paper introduces VERMEI, a new semantic testing method for FPGA logic synthesis tools that generates complex, equivalent program variants to effectively identify bugs and improve tool reliability.

Contribution

The paper presents VERMEI, a novel mutation-based testing approach that enhances bug detection in FPGA synthesis tools by generating complex, equivalent variants using Bayesian sampling.

Findings

VERMEI outperforms existing methods in bug detection.

Reported 15 bugs to vendors, with 9 confirmed as new.

Successfully identified complex bugs in FPGA synthesis tools.

Abstract

FPGA (Field-Programmable Gate Array) logic synthesis tools are key components in the EDA (Electronic Design Automation) toolchain. They convert hardware designs written in description languages such as Verilog into gate-level representations for FPGAs. However, defects in these tools may lead to unexpected behaviors and pose security risks. Therefore, it is crucial to harden these tools through testing. Although several methods have been proposed to automatically test FPGA logic synthesis tools, the challenge remains of insufficient semantic and logical complexity in test programs. In this paper, we propose VERMEI, a new method for testing FPGA logic synthesis tools. VERMEI consists of three modules: preprocessing, equivalent mutation, and bug identification. The preprocessing module identifies zombie logic (inactive code with no impact on the circuit output) in seed programs through simulation and coverage analysis. The equivalent mutation module generates equivalent variants of seed programs by pruning or inserting logic fragments in zombie areas. It uses Bayesian sampling to extract logic fragments from historical Verilog designs, making the generated variants have complex control flows and structures. The bug identification module, based on differential testing, compares the synthesized outputs of seed and variant programs to identify bugs. Experiments on Yosys, Vivado, and Quartus demonstrate that VERMEI outperforms the state-of-the-art methods. Within five months, VERMEI reported 15 bugs to vendors, 9 of which were confirmed as new.