Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

TL;DR

This survey reviews recent progress in NLP driven by large pre-trained transformer models, covering methods like fine-tuning, prompting, data augmentation, and discussing current limitations and future directions.

Contribution

It provides a comprehensive overview of recent techniques and applications of large pre-trained language models in NLP, highlighting new methods and research trends.

Findings

Large pre-trained models significantly improve NLP performance.

Various approaches like fine-tuning, prompting, and data augmentation are effective.

Identifies current limitations and future research directions.

Abstract

Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.