Parametric Knowledge and Retrieval Behavior in RAG Fine-Tuning for Electronic Design Automation

TL;DR

This paper evaluates RAG fine-tuning for long-form text generation in electronic design automation, introducing a new evaluation pipeline and metric that better capture factual accuracy and internalized knowledge.

Contribution

It presents TriFEX, a triple-based human-validated evaluation pipeline, and Parametric Knowledge Precision (PKP), a metric isolating internalized knowledge, revealing limitations of existing metrics.

Findings

ROUGE and BERTScore fail to detect factual differences.

PKP effectively measures internalized knowledge accuracy.

Smaller models outperform larger baselines on key metrics.

Abstract

Retrieval-Augmented Generation (RAG) fine-tuning has shown substantial improvements over vanilla RAG, yet most studies target document question answering and often rely on standard NLP metrics that can obscure factual differences. We evaluate RAG fine-tuning for long-form text generation in electronic design automation, adapting a 7B model under five context augmentation strategies with varying retrieval conditions. We introduce TriFEX, a human-validated, triple-based evaluation pipeline that attributes generated claims to their origin-user query, context and reference-and propose Parametric Knowledge Precision (PKP), which isolates internalized knowledge by filtering out claims leaked in the prompt. We show that ROUGE and BERTScore fail to detect factual differences that our triple-based evaluation reveals. Additionally, we demonstrate that an existing metric for knowledge internalization is retrieva-sensitive, with about 75% of its cross-condition variance driven by changes in the rate at which internal knowledge is expressed (PR), rather than by changes in its actual correctness (PKP). The fine-tuned 7B variants outperform a 72B baseline on most metrics, further showing generalization across conditions and on a related benchmark. These results underscore the limitations of available metrics in RAG evaluation and show that smaller models could be reasonably well adapted to specialized tasks for cost-efficient, on-premises deployment.