DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models

TL;DR

This paper introduces DSGram, an innovative evaluation framework for grammatical error correction that combines multiple metrics with dynamic weighting, leveraging large language models and human annotations to improve assessment accuracy.

Contribution

We propose DSGram, a novel evaluation method that integrates multiple criteria with a dynamic weighting mechanism using AHP and LLMs, addressing limitations of traditional reference-based metrics.

Findings

Enhanced evaluation accuracy for GEC models

Effective integration of semantic, edit, and fluency metrics

Validated with human annotations and LLM-simulated data

Abstract

Evaluating the performance of Grammatical Error Correction (GEC) models has become increasingly challenging, as large language model (LLM)-based GEC systems often produce corrections that diverge from provided gold references. This discrepancy undermines the reliability of traditional reference-based evaluation metrics. In this study, we propose a novel evaluation framework for GEC models, DSGram, integrating Semantic Coherence, Edit Level, and Fluency, and utilizing a dynamic weighting mechanism. Our framework employs the Analytic Hierarchy Process (AHP) in conjunction with large language models to ascertain the relative importance of various evaluation criteria. Additionally, we develop a dataset incorporating human annotations and LLM-simulated sentences to validate our algorithms and fine-tune more cost-effective models. Experimental results indicate that our proposed approach enhances the effectiveness of GEC model evaluations.