Improving LLM-Powered EDA Assistants with RAFT

TL;DR

This paper introduces RAFT, a method using synthetic question/answer data to improve large language models for electronic design automation tasks, addressing domain-specific knowledge gaps and enhancing retrieval-augmented generation performance.

Contribution

The paper proposes a novel RAFT approach utilizing synthetic data for fine-tuning LLMs in EDA, along with methods for secure access control and risk assessment of data leakage.

Findings

RAFT with synthetic data significantly improves LLM performance in EDA tasks.

Using real user questions as examples enhances synthetic data quality.

Implementing access control and evaluating data leakage risks ensures secure and reliable deployment.

Abstract

Electronic design engineers often struggle to efficiently access relevant information for tasks like design verification and technology development. While large language models (LLMs) can enhance productivity as conversational agents, pre-trained open-source LLMs lack domain-specific knowledge for Electronic Design Automation (EDA). In a Retrieval-Augmented Generation (RAG) context, LLMs rely on external context but may still produce inaccurate responses. Retrieval-Augmented Fine-Tuning (RAFT) improves LLM performance, but acquiring labeled question/answer (Q/A) data in EDA is difficult. To address this, we propose using synthetic Q/A datasets to enhance LLMs with RAFT. Our results show that RAFT with synthetic data significantly boosts LLM performance for RAG-based EDA tasks. We also investigate the impact of using real user questions as Retrieval-Augmented Few-Shot (RAFS) examples for synthetic data generation. Additionally, we implement secure access control to ensure sensitive information is only accessible to authorized personnel. Finally, we assess the risk of data leakage and unintended memorization during fine-tuning with synthetic data, providing practical insights.