TOWER: Tree Organized Weighting for Evaluating Complex Instructions

TL;DR

TOWER is a new evaluation metric for large language models that weights complex instruction following based on human-judged importance, improving assessment accuracy over existing benchmarks.

Contribution

Introducing TOWER, a tree-organized weighting method that incorporates human importance judgments into evaluating complex instruction adherence in LLMs.

Findings

Human annotators agree with tree-based instruction representations

TOWER improves evaluation accuracy for complex instructions

Tree annotations of InFoBench dataset are released for research use

Abstract

Evaluating the ability of large language models (LLMs) to follow complex human-written instructions is essential for their deployment in real-world applications. While benchmarks like Chatbot Arena use human judges to assess model performance, they are resource-intensive and time-consuming. Alternative methods using LLMs as judges, such as AlpacaEval, MT Bench, WildBench, and InFoBench offer improvements but still do not capture that certain complex instruction aspects are more important than others to follow. To address this gap, we propose a novel evaluation metric, \textsc{TOWER}, that incorporates human-judged importance into the assessment of complex instruction following. We show that human annotators agree with tree-based representations of these complex instructions nearly as much as they agree with other human annotators. We release tree-based annotations of the InFoBench dataset and the corresponding evaluation code to facilitate future research.