The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition

TL;DR

This paper demonstrates that open-source vision LLMs lack hierarchical knowledge of biological taxonomies, which hampers their ability for hierarchical visual recognition, and shows that fine-tuning on VQA tasks can improve hierarchical consistency.

Contribution

It reveals the hierarchical knowledge gap in open-source vision LLMs and shows that fine-tuning on VQA tasks enhances their hierarchical visual recognition capabilities.

Findings

LLMs lack awareness of biological taxonomies.

Fine-tuning improves hierarchical consistency.

VQA tasks reveal LLMs' bottleneck effect.

Abstract

This paper reveals that many open-source large language models (LLMs) lack hierarchical knowledge about our visual world, unaware of even well-established biology taxonomies. This shortcoming makes LLMs a bottleneck for vision LLMs' hierarchical visual recognition (e.g., recognizing Anemone Fish but not Vertebrate). We arrive at these findings using about one million four-choice visual question answering (VQA) tasks constructed from six taxonomies and four image datasets. Interestingly, finetuning a vision LLM using our VQA tasks reaffirms LLMs' bottleneck effect because the VQA tasks improve the LLMs' hierarchical consistency more than the vision LLMs'. We conjecture that one cannot make open-source vision LLMs understand visual concepts hierarchically until LLMs possess corresponding taxonomy knowledge.