A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks

TL;DR

This paper systematically evaluates prominent large language models on natural language generation tasks across English and Chinese datasets, highlighting their strengths and limitations in dialogue and summarization tasks.

Contribution

It introduces a comprehensive evaluation framework for LLMs in NLG tasks, including a unified setting with input templates and post-processing, and provides detailed analysis of model performances.

Findings

ChatGPT and T5 outperform others in dialogue generation.

LLaMA-based models show strong summarization capabilities.

Evaluation framework improves comparability across models.

Abstract

Recent efforts have evaluated large language models (LLMs) in areas such as commonsense reasoning, mathematical reasoning, and code generation. However, to the best of our knowledge, no work has specifically investigated the performance of LLMs in natural language generation (NLG) tasks, a pivotal criterion for determining model excellence. Thus, this paper conducts a comprehensive evaluation of well-known and high-performing LLMs, namely ChatGPT, ChatGLM, T5-based models, LLaMA-based models, and Pythia-based models, in the context of NLG tasks. We select English and Chinese datasets encompassing Dialogue Generation and Text Summarization. Moreover, we propose a common evaluation setting that incorporates input templates and post-processing strategies. Our study reports both automatic results, accompanied by a detailed analysis.