SafeFix: Targeted Model Repair via Controlled Image Generation

TL;DR

SafeFix introduces a novel approach for repairing visual recognition models by generating targeted, semantically faithful images for failure cases using controlled image generation and filtering, leading to improved robustness.

Contribution

It presents a new model repair method leveraging interpretable failure attribution and conditional image generation to address systematic errors in visual models.

Findings

Reduces errors on rare failure cases significantly.

Maintains model accuracy and avoids introducing new bugs.

Enhances robustness through targeted synthetic data augmentation.

Abstract

Deep learning models for visual recognition often exhibit systematic errors due to underrepresented semantic subpopulations. Although existing debugging frameworks can pinpoint these failures by identifying key failure attributes, repairing the model effectively remains difficult. Current solutions often rely on manually designed prompts to generate synthetic training images -- an approach prone to distribution shift and semantic errors. To overcome these challenges, we introduce a model repair module that builds on an interpretable failure attribution pipeline. Our approach uses a conditional text-to-image model to generate semantically faithful and targeted images for failure cases. To preserve the quality and relevance of the generated samples, we further employ a large vision-language model (LVLM) to filter the outputs, enforcing alignment with the original data distribution and maintaining semantic consistency. By retraining vision models with this rare-case-augmented synthetic dataset, we significantly reduce errors associated with rare cases. Our experiments demonstrate that this targeted repair strategy improves model robustness without introducing new bugs. Code is available at https://github.com/oxu2/SafeFix