AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

TL;DR

AutoChecklist introduces a modular, open-source framework for generating and scoring checklists with LLMs, enhancing interpretability and alignment in evaluation, with validated effectiveness and domain adaptability.

Contribution

The paper presents AutoChecklist, a novel library that unifies checklist-based evaluation into composable pipelines with a flexible taxonomy and supports multiple LLM providers.

Findings

Checklist methods align well with human preferences.

AutoChecklist's pipelines improve evaluation consistency.

The framework supports domain-specific adaptation.

Abstract

Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator $\rightarrow$ Refiner $\rightarrow$ Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.