Question answering system of bridge design specification based on large language model

TL;DR

This paper develops a question answering system for bridge design specifications using large language models, demonstrating high accuracy with full fine-tuning of BERT and exploring alternative training methods.

Contribution

It introduces three implementation schemes for a domain-specific QA system and evaluates their performance, highlighting the effectiveness of full fine-tuning of BERT.

Findings

Full fine-tuning of BERT achieves 100% accuracy on all datasets.

Parameter-efficient fine-tuning and scratch models need improved generalization.

The system effectively extracts answers from bridge design specifications.

Abstract

This paper constructs question answering system for bridge design specification based on large language model. Three implementation schemes are tried: full fine-tuning of the Bert pretrained model, parameter-efficient fine-tuning of the Bert pretrained model, and self-built language model from scratch. Through the self-built question and answer task dataset, based on the tensorflow and keras deep learning platform framework, the model is constructed and trained to predict the start position and end position of the answer in the bridge design specification given by the user. The experimental results show that full fine-tuning of the Bert pretrained model achieves 100% accuracy in the training-dataset, validation-dataset and test-dataset, and the system can extract the answers from the bridge design specification given by the user to answer various questions of the user; While parameter-efficient fine-tuning of the Bert pretrained model and self-built language model from scratch perform well in the training-dataset, their generalization ability in the test-dataset needs to be improved. The research of this paper provides a useful reference for the development of question answering system in professional field.