A Highly Efficient Diversity-based Input Selection for DNN Improvement Using VLMs

TL;DR

This paper introduces Concept-Based Diversity (CBD), an efficient image input selection metric leveraging Vision-Language Models, which improves DNN performance while reducing computational costs compared to existing methods.

Contribution

The paper proposes CBD, a novel, scalable diversity metric based on VLMs, and demonstrates its effectiveness in enhancing DNNs through hybrid selection with uncertainty measures.

Findings

CBD correlates strongly with Geometric Diversity but is computationally cheaper.

CBD-based selection outperforms state-of-the-art baselines in improving DNNs.

The approach remains efficient even on large datasets like ImageNet.

Abstract

Maintaining or improving the performance of Deep Neural Networks (DNNs) through fine-tuning requires labeling newly collected inputs, a process that is often costly and time-consuming. To alleviate this problem, input selection approaches have been developed in recent years to identify small, yet highly informative subsets for labeling. Diversity-based selection is one of the most effective approaches for this purpose. However, they are often computationally intensive and lack scalability for large input sets, limiting their practical applicability. To address this challenge, we introduce Concept-Based Diversity (CBD), a highly efficient metric for image inputs that leverages Vision-Language Models (VLM). Our results show that CBD exhibits a strong correlation with Geometric Diversity (GD), an established diversity metric, while requiring only a fraction of its computation time. Building on this finding, we propose a hybrid input selection approach that combines CBD with Margin, a simple uncertainty metric. We conduct a comprehensive evaluation across a diverse set of DNN models, input sets, selection budgets, and five most effective state-of-the-art selection baselines. The results demonstrate that the CBD-based selection consistently outperforms all baselines at guiding input selection to improve the DNN model. Furthermore, the CBD-based selection approach remains highly efficient, requiring selection times close to those of simple uncertainty-based methods such as Margin, even on larger input sets like ImageNet. These results confirm not only the effectiveness and computational advantage of the CBD-based approach, particularly compared to hybrid baselines, but also its scalability in repetitive and extensive input selection scenarios.