Beyond Binary Correctness: Scaling Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks

TL;DR

This paper introduces LH-Bench, a novel evaluation framework for long-horizon, subjective enterprise tasks that incorporates expert rubrics, curated artifacts, and human preferences to reliably assess AI performance beyond binary correctness.

Contribution

The paper presents LH-Bench, a three-pillar evaluation method that enables scalable, reliable assessment of long-term, subjective enterprise AI tasks using expert-grounded rubrics and human preferences.

Findings

Expert-grounded rubrics improve evaluation reliability (kappa=0.60).

Human preferences validate rubric-based assessments (p<0.05).

Public datasets for Figma-to-code and content tasks are released.

Abstract

Large language models excel on objectively verifiable tasks such as math and programming, where evaluation reduces to unit tests or a single correct answer. In contrast, real-world enterprise work is often subjective and context-dependent: success hinges on organizational goals, user intent, and the quality of intermediate artifacts produced across long, multi-tool workflows. We introduce LH-Bench, a three-pillar evaluation design that moves beyond binary correctness to score autonomous, long-horizon execution on subjective enterprise tasks. The pillars are: (i) expert-grounded rubrics that give LLM judges the domain context needed to score subjective work, (ii) curated ground-truth artifacts that enable stepwise reward signals (e.g., chapter-level annotation for content tasks), and (iii) pairwise human preference evaluation for convergent validation. We show that domain-authored rubrics provide substantially more reliable evaluation signals than LLM-authored rubrics (kappa = 0.60 vs. 0.46), and that human preference judgments confirm the same top-tier separation (p < 0.05), evidence that expert-grounded evaluation can scale without sacrificing reliability. We release public datasets and report results on two environments: Figma-to-code (33 real .fig tasks against the Figma API via MCP) and Programmatic content (41 courses comprising 183 individually-evaluated chapters on a course platform serving 30+ daily users).