Learning Consistent Taxonomic Classification through Hierarchical Reasoning

TL;DR

This paper introduces VL-Taxon, a hierarchical reasoning framework that significantly improves taxonomic classification accuracy and consistency in Vision-Language Models by employing a two-stage process with supervised and reinforcement learning.

Contribution

The paper presents a novel two-stage hierarchical reasoning framework, VL-Taxon, that enhances both leaf-level accuracy and hierarchical consistency in VLMs, trained with minimal data and no reliance on other VLM-generated examples.

Findings

VL-Taxon outperforms baseline models by over 10% in accuracy.

The framework improves hierarchical consistency in taxonomic classification.

Effective with limited training data without using other VLM-generated examples.

Abstract

While Vision-Language Models (VLMs) excel at visual understanding, they often fail to grasp hierarchical knowledge. This leads to common errors where VLMs misclassify coarser taxonomic levels even when correctly identifying the most specific level (leaf level). Existing approaches largely overlook this issue by failing to model hierarchical reasoning. To address this gap, we propose VL-Taxon, a two-stage, hierarchy-based reasoning framework designed to improve both leaf-level accuracy and hierarchical consistency in taxonomic classification. The first stage employs a top-down process to enhance leaf-level classification accuracy. The second stage then leverages this accurate leaf-level output to ensure consistency throughout the entire taxonomic hierarchy. Each stage is initially trained with supervised fine-tuning to instill taxonomy knowledge, followed by reinforcement learning to refine the model's reasoning and generalization capabilities. Extensive experiments reveal a remarkable result: our VL-Taxon framework, implemented on the Qwen2.5-VL-7B model, outperforms its original 72B counterpart by over 10% in both leaf-level and hierarchical consistency accuracy on average on the iNaturalist-2021 dataset. Notably, this significant gain was achieved by fine-tuning on just a small subset of data, without relying on any examples generated by other VLMs.