LockForge: Automating Paper-to-Code for Logic Locking with Multi-Agent Reasoning LLMs

TL;DR

LockForge is a novel multi-agent LLM framework that automates turning logic locking research papers into executable code, enhancing reproducibility and systematic validation of complex schemes.

Contribution

It introduces a multi-stage pipeline with LLM-based validation stages, enabling automatic code generation and assessment for logic locking schemes from academic descriptions.

Findings

Effective code generation for 10 seminal LL schemes

Multi-stage validation improves reproducibility and correctness

Advanced reasoning models are essential for complex tasks

Abstract

Despite rapid progress in logic locking (LL), reproducibility remains a challenge as codes are rarely made public. We present LockForge, a first-of-its-kind, multi-agent large language model (LLM) framework that turns LL descriptions in papers into executable and tested code. LockForge provides a carefully crafted pipeline realizing forethought, implementation, iterative refinement, and a multi-stage validation, all to systematically bridge the gap between prose and practice for complex LL schemes. For validation, we devise (i) an LLM-as-Judge stage with a scoring system considering behavioral checks, conceptual mechanisms, structural elements, and reproducibility on benchmarks, and (ii) an independent LLM-as-Examiner stage for ground-truth assessment. We apply LockForge to 10 seminal LL schemes, many of which lack reference implementations. Our evaluation on multiple SOTA LLMs, including ablation studies, reveals the significant complexity of the task. We show that an advanced reasoning model and a sophisticated, multi-stage framework like LockForge are required. We release all implementations and benchmarks, providing a reproducible and fair foundation for evaluation of further LL research.