Evaluating Multi-turn Human-AI Interaction

TL;DR

This paper critiques current NLP evaluation methods for human-AI interactions and introduces TCR, a structured framework emphasizing transparency, consistency, and refinement for better human-centered assessment.

Contribution

It proposes a novel evaluation framework, TCR, tailored for human-AI interaction, with structured prompts and examples to complement traditional metrics.

Findings

TCR highlights critical interaction dimensions like transparency and consistency.

Structured evaluation prompts improve assessment of multi-turn interactions.

The framework demonstrates how to complement aggregate metrics with human-centered evaluation.

Abstract

Large language models (LLMs) are increasingly used as collaborative assistants, yet dominant NLP evaluation practices remain centered on aggregate metrics such as accuracy and fluency. These approaches often overlook behaviors that are critical in human-facing settings (e.g., consistency across multiple turns and iterative refinement). In this paper, we examine limitations of current NLP evaluation practices and introduce TCR, a structured framework for evaluating human--AI interaction using educational LLM assistants as an illustrative example. TCR emphasizes dimensions such as transparency, consistency, and refinement. We further present structured evaluation prompts and illustrative interaction examples demonstrating how structured evaluation can complement aggregate metrics and LLM-as-a-judge approaches. Our work highlights the need for more human-centered evaluation practices for interactive LLM systems.