Evaluating Text Summaries Generated by Large Language Models Using OpenAI's GPT

TL;DR

This study explores using OpenAI's GPT models as independent evaluators of text summaries, comparing its assessments with traditional metrics to determine its effectiveness in measuring summary quality.

Contribution

It introduces GPT as a novel, independent evaluator for text summaries, providing a new approach that complements traditional evaluation metrics.

Findings

GPT evaluations correlate well with traditional metrics

GPT effectively assesses relevance and coherence

Potential for GPT to improve summary evaluation processes

Abstract

This research examines the effectiveness of OpenAI's GPT models as independent evaluators of text summaries generated by six transformer-based models from Hugging Face: DistilBART, BERT, ProphetNet, T5, BART, and PEGASUS. We evaluated these summaries based on essential properties of high-quality summary - conciseness, relevance, coherence, and readability - using traditional metrics such as ROUGE and Latent Semantic Analysis (LSA). Uniquely, we also employed GPT not as a summarizer but as an evaluator, allowing it to independently assess summary quality without predefined metrics. Our analysis revealed significant correlations between GPT evaluations and traditional metrics, particularly in assessing relevance and coherence. The results demonstrate GPT's potential as a robust tool for evaluating text summaries, offering insights that complement established metrics and providing a basis for comparative analysis of transformer-based models in natural language processing tasks.