Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation

TL;DR

This paper presents InteractEval, a framework combining human expertise and LLMs using the Think-Aloud method to improve checklist-based text evaluation across multiple quality dimensions.

Contribution

It introduces a novel framework that integrates human and LLM reasoning with the Think-Aloud method, enhancing attribute generation and evaluation performance.

Findings

Humans excel at internal quality attributes like Coherence and Fluency.

LLMs perform better on external alignment attributes like Consistency and Relevance.

Combining humans and LLMs yields the best evaluation outcomes.

Abstract

This study introduces \textbf{InteractEval}, a framework that integrates human expertise and Large Language Models (LLMs) using the Think-Aloud (TA) method to generate attributes for checklist-based text evaluation. By combining human flexibility and reasoning with LLM consistency, InteractEval outperforms traditional non-LLM-based and LLM-based baselines across four distinct dimensions, consisting of Coherence, Fluency, Consistency, and Relevance. The experiment also investigates the effectiveness of the TA method, showing that it promotes divergent thinking in both humans and LLMs, leading to the generation of a wider range of relevant attributes and enhance text evaluation performance. Comparative analysis reveals that humans excel at identifying attributes related to internal quality (Coherence and Fluency), but LLMs perform better at those attributes related to external alignment (Consistency and Relevance). Consequently, leveraging both humans and LLMs together produces the best evaluation outcomes. In other words, this study emphasizes the necessity of effectively combining humans and LLMs in an automated checklist-based text evaluation framework. The code is available at \textbf{\url{https://github.com/BBeeChu/InteractEval.git}}.