Parachute: Evaluating Interactive Human-LM Co-writing Systems

TL;DR

This paper introduces Parachute, a comprehensive evaluation framework for assessing interactive human-LM co-writing systems, addressing the lack of systematic evaluation methods in this emerging area.

Contribution

The paper presents Parachute, a novel human-centered evaluation framework with categorized metrics for assessing interactive co-writing systems involving language models.

Findings

Demonstrates how to evaluate co-writing systems using Parachute

Provides a structured approach to compare different systems

Highlights the importance of interaction evaluation in co-writing

Abstract

A surge of advances in language models (LMs) has led to significant interest in using LMs to build co-writing systems, in which humans and LMs interactively contribute to a shared writing artifact. However, there is a lack of studies assessing co-writing systems in interactive settings. We propose a human-centered evaluation framework, Parachute, for interactive co-writing systems. Parachute showcases an integrative view of interaction evaluation, where each evaluation aspect consists of categorized practical metrics. Furthermore, we present Parachute with a use case to demonstrate how to evaluate and compare co-writing systems using Parachute.