Evaluating Human-Language Model Interaction

TL;DR

This paper introduces HALIE, a comprehensive framework for evaluating human-LM interaction that considers the interactive process, subjective experience, and preferences, revealing that non-interactive metrics often do not predict interactive performance.

Contribution

The paper develops HALIE, a novel evaluation framework for human-LM interaction, and demonstrates its effectiveness across diverse tasks and models, highlighting discrepancies with traditional non-interactive metrics.

Findings

Better non-interactive performance does not always mean better human-LM interaction

Interactive and non-interactive metrics can diverge in evaluation results

Human-centric evaluation captures aspects missed by standard benchmarks

Abstract

Many real-world applications of language models (LMs), such as writing assistance and code autocomplete, involve human-LM interaction. However, most benchmarks are non-interactive in that a model produces output without human involvement. To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics. Compared to standard, non-interactive evaluation, HALIE captures (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality (e.g., enjoyment and ownership). We then design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. With four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21 Labs' Jurassic-1), we find that better non-interactive performance does not always translate to better human-LM interaction. In particular, we highlight three cases where the results from non-interactive and interactive metrics diverge and underscore the importance of human-LM interaction for LM evaluation.