FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications

TL;DR

FireBench is a comprehensive benchmark designed to evaluate how well large language models follow instructions in enterprise and API-driven contexts, addressing a gap in existing evaluation methods.

Contribution

The paper introduces FireBench, a new benchmark grounded in real-world enterprise use cases, covering six capability dimensions with over 2,400 samples, and evaluates 11 LLMs.

Findings

LLMs vary significantly in instruction following in enterprise scenarios.

FireBench reveals specific strengths and weaknesses of models across different capabilities.

Open-sourcing FireBench facilitates community-driven improvements and model diagnostics.

Abstract

Instruction following is critical for LLMs deployed in enterprise and API-driven settings, where strict adherence to output formats, content constraints, and procedural requirements is essential for enabling reliable LLM-assisted workflows. However, existing instruction following benchmarks predominantly evaluate natural language generation constraints that reflect the needs of chat assistants rather than enterprise users. To bridge this gap, we introduce FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns. FireBench evaluates six core capability dimensions across diverse applications including information extraction, customer support, and coding agents, comprising over 2,400 samples. We evaluate 11 LLMs and present key findings on their instruction following behavior in enterprise scenarios. We open-source FireBench at fire-bench.com to help users assess model suitability, support model developers in diagnosing performance, and invite community contributions.