VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labels

TL;DR

VISTA is a visual analytics framework designed to improve the quality of labels generated by foundation models for open-vocabulary image segmentation, combining multi-phased validation and human expertise.

Contribution

It introduces a comprehensive visual analytics approach to validate and correct FM-generated labels, addressing the gap in quality assessment for large-scale datasets.

Findings

VISTA effectively identifies issues in FM-generated labels.

Human-in-the-loop validation improves label quality.

Demonstrated success on benchmark datasets.

Abstract

The advances in multi-modal foundation models (FMs) (e.g., CLIP and LLaVA) have facilitated the auto-labeling of large-scale datasets, enhancing model performance in challenging downstream tasks such as open-vocabulary object detection and segmentation. However, the quality of FM-generated labels is less studied as existing approaches focus more on data quantity over quality. This is because validating large volumes of data without ground truth presents a considerable challenge in practice. Existing methods typically rely on limited metrics to identify problematic data, lacking a comprehensive perspective, or apply human validation to only a small data fraction, failing to address the full spectrum of potential issues. To overcome these challenges, we introduce VISTA, a visual analytics framework that improves data quality to enhance the performance of multi-modal models. Targeting the complex and demanding domain of open-vocabulary image segmentation, VISTA integrates multi-phased data validation strategies with human expertise, enabling humans to identify, understand, and correct hidden issues within FM-generated labels. Through detailed use cases on two benchmark datasets and expert reviews, we demonstrate VISTA's effectiveness from both quantitative and qualitative perspectives.