HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing

TL;DR

This paper introduces Tree-of-Writing (ToW), a hierarchical evaluation method for large language models' writing, and presents HowToBench, a comprehensive Chinese writing benchmark, demonstrating ToW's superior correlation with human judgments and robustness.

Contribution

The paper proposes ToW, a tree-structured evaluation framework, and introduces HowToBench, a large-scale Chinese writing benchmark for assessing LLMs across multiple genres and tasks.

Findings

ToW achieves 0.93 Pearson correlation with human judgments.

ToW is robust against textual disturbances affecting other metrics.

Input length negatively correlates with content scores in the Guide task.

Abstract

Evaluating the writing capabilities of large language models (LLMs) remains a significant challenge due to the multidimensional nature of writing skills and the limitations of existing metrics. LLM's performance in thousand-words level and open-ended writing is inadequately assessed by traditional reference-based metrics or modern LLM-as-a-judge methods. We propose Tree-of-Writing (ToW), to resolve the implicit inconsistency often found when LLM-as-a-judge aggregates all sub-features in text evaluation. ToW incorporates a tree-structured workflow by explicitly modeling the aggregation weights of sub-features. We also present HowToBench, a large-scale Chinese writing benchmark encompassing 12 genres and 1302 instructions across three task categories: contextual completion, outline-guided writing, and open-ended generation. ToW successfully mitigates the biases, achieving a 0.93 Pearson correlation with human judgments. Furthermore, we detect that both overlap-based text generation metrics and popular LLM-as-a-judge practices are vulnerable to textual disturbances, while ToW is robust to them. We also uncover a negative correlation between input length and content-related scores in the Guide task, showcasing that it cannot be simply improved by input-side information piling.