Performance of Transfer Learning Model vs. Traditional Neural Network in Low System Resource Environment

TL;DR

This paper compares the performance and resource efficiency of lightweight transfer learning models versus traditional neural networks in low-resource environments for NLP tasks like text classification and NER.

Contribution

It provides an analysis of the trade-offs between transfer learning models and traditional neural networks under limited computational resources.

Findings

Lighter transfer learning models require fewer resources than large pre-trained models.

Traditional neural networks can be more efficient in extremely low-resource settings.

Transfer learning models generally achieve higher accuracy with moderate resource use.

Abstract

Recently, the use of pre-trained model to build neural network based on transfer learning methodology is increasingly popular. These pre-trained models present the benefit of using less computing resources to train model with smaller amount of training data. The rise of state-of-the-art models such as BERT, XLNet and GPT boost accuracy and benefit as a base model for transfer leanring. However, these models are still too complex and consume many computing resource to train for transfer learning with low GPU memory. We will compare the performance and cost between lighter transfer learning model and purposely built neural network for NLP application of text classification and NER model.