An Automated Knowledge Mining and Document Classification System with Multi-model Transfer Learning

TL;DR

This paper introduces an automated system for knowledge mining and document classification that leverages multi-model transfer learning with techniques like fine-tuning, pruning, and multi-model training to improve accuracy over baseline methods.

Contribution

It proposes a novel multi-model transfer learning approach with fine-tuning, pruning, and multiple BERT models to enhance document classification performance.

Findings

Outperforms baseline BERT and BERT-CNN methods in accuracy.

Uses multi-model training to reduce randomness in fine-tuning.

Achieves higher MCC scores on the CoLA dataset.

Abstract

Service manual documents are crucial to the engineering company as they provide guidelines and knowledge to service engineers. However, it has become inconvenient and inefficient for service engineers to retrieve specific knowledge from documents due to the complexity of resources. In this research, we propose an automated knowledge mining and document classification system with novel multi-model transfer learning approaches. Particularly, the classification performance of the system has been improved with three effective techniques: fine-tuning, pruning, and multi-model method. The fine-tuning technique optimizes a pre-trained BERT model by adding a feed-forward neural network layer and the pruning technique is used to retrain the BERT model with new data. The multi-model method initializes and trains multiple BERT models to overcome the randomness of data ordering during the fine-tuning process. In the first iteration of the training process, multiple BERT models are being trained simultaneously. The best model is then selected for the next phase of the training process with another two iterations and the training processes for other BERT models will be terminated. The performance of the proposed system has been evaluated by comparing with two robust baseline methods, BERT and BERT-CNN. Experimental results on a widely used Corpus of Linguistic Acceptability (CoLA) dataset have shown that the proposed techniques perform better than these baseline methods in terms of accuracy and MCC score.