Few-shot learning approaches for classifying low resource domain specific software requirements

TL;DR

This paper investigates few-shot learning methods to classify low-resource automotive software requirements using pre-trained NLP models, demonstrating that certain models perform well with minimal annotated data.

Contribution

It evaluates multiple algorithms for fine-tuning pre-trained models on automotive domain requirements with only 15 samples per category, highlighting effective approaches in low-resource settings.

Findings

SciBERT and DeBERTa outperform others at 15 samples

Performance gains plateau beyond 50 samples for some models

Siamese and T5 models show competitive results with fewer samples

Abstract

With the advent of strong pre-trained natural language processing models like BERT, DeBERTa, MiniLM, T5, the data requirement for industries to fine-tune these models to their niche use cases has drastically reduced (typically to a few hundred annotated samples for achieving a reasonable performance). However, the availability of even a few hundred annotated samples may not always be guaranteed in low resource domains like automotive, which often limits the usage of such deep learning models in an industrial setting. In this paper we aim to address the challenge of fine-tuning such pre-trained models with only a few annotated samples, also known as Few-shot learning. Our experiments focus on evaluating the performance of a diverse set of algorithms and methodologies to achieve the task of classifying BOSCH automotive domain textual software requirements into 3 categories, while utilizing only 15 annotated samples per category for fine-tuning. We find that while SciBERT and DeBERTa based models tend to be the most accurate at 15 training samples, their performance improvement scales minimally as the number of annotated samples is increased to 50 in comparison to Siamese and T5 based models.