A Lightweight Multi-Expert Generative Language Model System for Engineering Information and Knowledge Extraction

TL;DR

This paper presents a lightweight, graph-based multi-expert language model system called SLG that improves engineering information extraction by reducing computational costs and hallucination issues, outperforming traditional fine-tuning methods.

Contribution

The introduction of the Small Language Graph (SLG), a novel lightweight, graph-structured adaptation method with small expert models for efficient and accurate engineering text generation.

Findings

SLG surpasses conventional fine-tuning on Exact Match by 3x.

Fine-tuning with SLG is 1.7x faster than larger models.

SLG enables cost-effective deployment for small to medium engineering firms.

Abstract

Despite recent advancements in domain adaptation techniques for large language models, these methods remain computationally intensive, and the resulting models can still exhibit hallucination issues. Most existing adaptation methods do not prioritize reducing the computational resources required for fine-tuning and inference of language models. Hallucination issues have gradually decreased with each new model release. However, they remain prevalent in engineering contexts, where generating well-structured text with minimal errors and inconsistencies is critical. This work introduces a novel approach called the Small Language Graph (SLG), which is a lightweight adaptation solution designed to address the two key challenges outlined above. The system is structured in the form of a graph, where each node represents a lightweight expert - a small language model fine-tuned on specific and concise texts. The results of this study have shown that SLG was able to surpass conventional fine-tuning methods on the Exact Match metric by 3 times. Additionally, the fine-tuning process was 1.7 times faster compared to that of a larger stand-alone language model. These findings introduce a potential for small to medium-sized engineering companies to confidently use generative AI technologies, such as LLMs, without the necessity to invest in expensive computational resources. Also, the graph architecture and the small size of expert nodes offer a possible opportunity for distributed AI systems, thus potentially diverting the global need for expensive centralized compute clusters.