LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers

TL;DR

LADDER leverages natural language and large language models to discover and rectify biases in vision classifiers, overcoming limitations of traditional methods by enabling complex reasoning and domain knowledge integration.

Contribution

The paper introduces LADDER, a novel language-driven approach that improves bias discovery and mitigation in vision classifiers without relying on explicit attribute annotations.

Findings

LADDER outperforms existing methods in bias discovery across 6 datasets.

It effectively mitigates biases without explicit attribute annotations.

Demonstrates robustness across diverse datasets and model architectures.

Abstract

Error slice discovery is crucial to diagnose and mitigate model errors. Current clustering or discrete attribute-based slice discovery methods face key limitations: 1) clustering results in incoherent slices, while assigning discrete attributes to slices leads to incomplete coverage of error patterns due to missing or insufficient attributes; 2) these methods lack complex reasoning, preventing them from fully explaining model biases; 3) they fail to integrate \textit{domain knowledge}, limiting their usage in specialized fields \eg radiology. We propose\ladder (\underline{La}nguage-\underline{D}riven \underline{D}iscovery and \underline{E}rror \underline{R}ectification), to address the limitations by: (1) leveraging the flexibility of natural language to address incompleteness, (2) employing LLM's latent \textit{domain knowledge} and advanced reasoning to analyze sentences and derive testable hypotheses directly, identifying biased attributes, and form coherent error slices without clustering. Existing mitigation methods typically address only the worst-performing group, often amplifying errors in other subgroups. In contrast,\ladder generates pseudo attributes from the discovered hypotheses to mitigate errors across all biases without explicit attribute annotations or prior knowledge of bias. Rigorous evaluations on 6 datasets spanning natural and medical images -- comparing 200+ classifiers with diverse architectures, pretraining strategies, and LLMs -- show that\ladder consistently outperforms existing baselines in discovering and mitigating biases.