Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification

TL;DR

AuditDM is an automated framework that identifies and rectifies capability gaps in multimodal large language models by actively discovering failure modes through divergence auditing, leading to improved model performance.

Contribution

The paper introduces AuditDM, a reinforcement learning-based auditing framework that uncovers diverse failure modes in MLLMs and enhances their capabilities through targeted rectification.

Findings

Discoveries of over 20 distinct failure types in state-of-the-art models.

Consistent performance improvements across 16 benchmarks after fine-tuning.

A 3B model surpasses a 28B model after targeted auditing and rectification.

Abstract

Conventional evaluation methods for multimodal LLMs (MLLMs) lack interpretability and are often insufficient to fully disclose significant capability gaps across models. To address this, we introduce AuditDM, an automated framework that actively discovers and rectifies MLLM failure modes by auditing their divergence. AuditDM fine-tunes an MLLM as an auditor via reinforcement learning to generate challenging questions and counterfactual images that maximize disagreement among target models. Once trained, the auditor uncovers diverse, interpretable exemplars that reveal model weaknesses and serve as annotation-free data for rectification. When applied to SoTA models like Gemma-3 and PaliGemma-2, AuditDM discovers more than 20 distinct failure types. Fine-tuning on these discoveries consistently improves all models across 16 benchmarks, and enables a 3B model to surpass its 28B counterpart. Our results suggest that as data scaling hits diminishing returns, targeted model auditing offers an effective path to model diagnosis and improvement.