Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments

TL;DR

DRAFT is a novel fine-tuning approach for large language models that enhances safety-critical software assessments by integrating dual document retrieval and a semi-automated dataset generation process.

Contribution

It introduces a new fine-tuning framework with dual-retrieval architecture and a semi-automated dataset creation method for improved compliance assessment.

Findings

GPT-4o-mini with DRAFT shows a 7% accuracy improvement.

Qualitative enhancements in evidence handling and reasoning.

Effective in complex regulatory compliance scenarios.

Abstract

Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents Document Retrieval-Augmented Fine-Tuning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework that accommodates our dual-retrieval architecture, which simultaneously accesses both software documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodology that incorporates variable numbers of relevant documents with meaningful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative improvements in evidence handling, response structure, and domain-specific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transparency and evidence-based reasoning essential in regulatory domains.