LVLM-Aided Alignment of Task-Specific Vision Models

TL;DR

This paper introduces LVLM-VA, a novel method that uses large vision language models to align small, task-specific vision models with human knowledge, reducing reliance on spurious features.

Contribution

The paper presents a new LVLM-aided approach for aligning vision models with human knowledge, improving interpretability and robustness without fine-grained feedback.

Findings

Significant improvement in model alignment with human specifications.

Reduces dependence on spurious features and biases.

Validated on synthetic and real-world datasets.

Abstract

In high-stakes domains, small task-specific vision models are crucial due to their low computational requirements and the availability of numerous methods to explain their results. However, these explanations often reveal that the models do not align well with human domain knowledge, relying instead on spurious correlations. This might result in brittle behavior once deployed in the real-world. To address this issue, we introduce a novel and efficient method for aligning small task-specific vision models with human domain knowledge by leveraging the generalization capabilities of a Large Vision Language Model (LVLM). Our LVLM-Aided Visual Alignment (LVLM-VA) method provides a bidirectional interface that translates model behavior into natural language and maps human class-level specifications to image-level critiques, enabling effective interaction between domain experts and the model. Our method demonstrates substantial improvement in aligning model behavior with human specifications, as validated on both synthetic and real-world datasets. We show that it effectively reduces the model's dependence on spurious features and on group-specific biases, without requiring fine-grained feedback.