Is ChatGPT a General-Purpose Natural Language Processing Task Solver?

TL;DR

This paper empirically evaluates ChatGPT's ability to perform a wide range of NLP tasks zero-shot, revealing its strengths in reasoning tasks and limitations in sequence tagging, thus assessing its potential as a general-purpose NLP solver.

Contribution

The study provides a comprehensive empirical analysis of ChatGPT's zero-shot performance across diverse NLP tasks, highlighting its capabilities and current limitations.

Findings

Performs well on reasoning tasks like arithmetic reasoning

Struggles with sequence tagging tasks

Shows both strengths and limitations in zero-shot NLP applications

Abstract

Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot -- i.e., without adaptation on downstream data. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community due to the fact that it can generate high-quality responses to human input and self-correct previous mistakes based on subsequent conversations. However, it is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot. In this work, we empirically analyze the zero-shot learning ability of ChatGPT by evaluating it on 20 popular NLP datasets covering 7 representative task categories. With extensive empirical studies, we demonstrate both the effectiveness and limitations of the current version of ChatGPT. We find that ChatGPT performs well on many tasks favoring reasoning capabilities (e.g., arithmetic reasoning) while it still faces challenges when solving specific tasks such as sequence tagging. We additionally provide in-depth analysis through qualitative case studies.