Text generation for dataset augmentation in security classification tasks

TL;DR

This paper explores using advanced language models like GPT-3 to generate synthetic data for improving security-related text classifiers, especially when positive samples are scarce, showing significant performance gains.

Contribution

It introduces novel GPT-3 based data augmentation methods tailored for security classification tasks and evaluates their effectiveness against existing strategies.

Findings

GPT-3 augmentation improves classifier performance

Significant benefits in low positive sample scenarios

Outperforms basic augmentation methods

Abstract

Security classifiers, designed to detect malicious content in computer systems and communications, can underperform when provided with insufficient training data. In the security domain, it is often easy to find samples of the negative (benign) class, and challenging to find enough samples of the positive (malicious) class to train an effective classifier. This study evaluates the application of natural language text generators to fill this data gap in multiple security-related text classification tasks. We describe a variety of previously-unexamined language-model fine-tuning approaches for this purpose and consider in particular the impact of disproportionate class-imbalances in the training set. Across our evaluation using three state-of-the-art classifiers designed for offensive language detection, review fraud detection, and SMS spam detection, we find that models trained with GPT-3 data augmentation strategies outperform both models trained without augmentation and models trained using basic data augmentation strategies already in common usage. In particular, we find substantial benefits for GPT-3 data augmentation strategies in situations with severe limitations on known positive-class samples.