InFoBench: Evaluating Instruction Following Ability in Large Language Models

TL;DR

This paper introduces a new metric, DRFR, and a benchmark, InFoBench, to evaluate large language models' ability to follow complex instructions more reliably and comprehensively.

Contribution

It presents DRFR, a decomposed requirement-following metric, and InFoBench, a diverse instruction benchmark, enhancing evaluation of LLMs' instruction-following capabilities.

Findings

DRFR outperforms traditional scoring methods in reliability.

GPT-4 is an effective, cost-efficient annotator.

Advanced LLMs show strengths and weaknesses in complex instructions.

Abstract

This paper introduces the Decomposed Requirements Following Ratio (DRFR), a new metric for evaluating Large Language Models' (LLMs) ability to follow instructions. Addressing a gap in current methodologies, DRFR breaks down complex instructions into simpler criteria, facilitating a detailed analysis of LLMs' compliance with various aspects of tasks. Alongside this metric, we present InFoBench, a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories. Our experiments compare DRFR with traditional scoring methods and explore annotation sources, including human experts, crowd-sourced workers, and GPT-4. The findings demonstrate DRFR's higher reliability and the effectiveness of using GPT-4 as a cost-efficient annotator. The evaluation of several advanced LLMs using this framework reveals their strengths and areas needing improvement, particularly in complex instruction-following. This study contributes a novel metric and benchmark, offering insights for future LLM development and evaluation.