CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification. VII Concurso de Teses, Disserta\c{c}\~oes e Trabalhos de Gradua\c{c}\~ao em SI -- XXI Simp\'osio Brasileiro de Sistemas de Informa\c{c}\~ao

TL;DR

This paper explores instance selection techniques for automatic text classification, proposing new methods that significantly reduce training data size and computational costs while maintaining model effectiveness, especially for large datasets and transformer models.

Contribution

It provides a comprehensive comparison of existing instance selection methods and introduces two novel noise- and redundancy-aware solutions tailored for large NLP datasets and transformer architectures.

Findings

Achieved an average 41% reduction in training data size without loss of effectiveness.

Demonstrated speedup improvements of up to 2.46x in training time.

Confirmed the untapped potential of instance selection in NLP tasks.

Abstract

Progress in Natural Language Processing (NLP) has been dictated by the rule of more: more data, more computing power and more complexity, best exemplified by the Large Language Models. However, training (or fine-tuning) large dense models for specific applications usually requires significant amounts of computing resources. This \textbf{Ph.D. dissertation} focuses on an under-investi\-gated NLP data engineering technique, whose potential is enormous in the current scenario known as Instance Selection (IS). The IS goal is to reduce the training set size by removing noisy or redundant instances while maintaining the effectiveness of the trained models and reducing the training process cost. We provide a comprehensive and scientifically sound comparison of IS methods applied to an essential NLP task -- Automatic Text Classification (ATC), considering several classification solutions and many datasets. Our findings reveal a significant untapped potential for IS solutions. We also propose two novel IS solutions that are noise-oriented and redundancy-aware, specifically designed for large datasets and transformer architectures. Our final solution achieved an average reduction of 41\% in training sets, while maintaining the same levels of effectiveness in all datasets. Importantly, our solutions demonstrated speedup improvements of 1.67x (up to 2.46x), making them scalable for datasets with hundreds of thousands of documents.