Towards Secure and Private Language Models for Nuclear Power Plants

TL;DR

This paper presents a small, secure, domain-specific language model for nuclear power plants, trained on a single GPU, demonstrating potential for specialized nuclear applications while highlighting areas for improvement.

Contribution

It introduces a compact Transformer-based LLM trained on nuclear data, emphasizing cybersecurity and data confidentiality for sensitive nuclear applications.

Findings

The model captures specialized nuclear vocabulary.

Generated text shows signs of domain relevance.

The approach is feasible for in-house nuclear LLMs.

Abstract

This paper introduces a domain-specific Large Language Model for nuclear applications, built from the publicly accessible Essential CANDU textbook. Drawing on a compact Transformer-based architecture, the model is trained on a single GPU to protect the sensitive data inherent in nuclear operations. Despite relying on a relatively small dataset, it shows encouraging signs of capturing specialized nuclear vocabulary, though the generated text sometimes lacks syntactic coherence. By focusing exclusively on nuclear content, this approach demonstrates the feasibility of in-house LLM solutions that align with rigorous cybersecurity and data confidentiality standards. Early successes in text generation underscore the model's utility for specialized tasks, while also revealing the need for richer corpora, more sophisticated preprocessing, and instruction fine-tuning to enhance domain accuracy. Future directions include extending the dataset to cover diverse nuclear subtopics, refining tokenization to reduce noise, and systematically evaluating the model's readiness for real-world applications in nuclear domain.