Improving Fine-grained Visual Understanding in VLMs through Text-Only Training

TL;DR

This paper explores using text-only training to improve fine-grained visual understanding in visual-language models, showing it can be as effective as traditional methods while reducing resource requirements.

Contribution

It demonstrates that text-only training enhances fine-grained visual recognition in VLMs, offering a resource-efficient alternative to image-text training.

Findings

Text-only training achieves comparable performance to image-text training.

Significant reduction in computational costs with text-only methods.

Effective across diverse domains like species classification and cultural understanding.

Abstract

Visual-Language Models (VLMs) have become a powerful tool for bridging the gap between visual and linguistic understanding. However, the conventional learning approaches for VLMs often suffer from limitations, such as the high resource requirements of collecting and training image-text paired data. Recent research has suggested that language understanding plays a crucial role in the performance of VLMs, potentially indicating that text-only training could be a viable approach. In this work, we investigate the feasibility of enhancing fine-grained visual understanding in VLMs through text-only training. Inspired by how humans develop visual concept understanding, where rich textual descriptions can guide visual recognition, we hypothesize that VLMs can also benefit from leveraging text-based representations to improve their visual recognition abilities. We conduct comprehensive experiments on two distinct domains: fine-grained species classification and cultural visual understanding tasks. Our findings demonstrate that text-only training can be comparable to conventional image-text training while significantly reducing computational costs. This suggests a more efficient and cost-effective pathway for advancing VLM capabilities, particularly valuable in resource-constrained environments.