Hierarchy-Aware Fine-Tuning of Vision-Language Models

TL;DR

This paper introduces a hierarchy-aware fine-tuning method for vision-language models that improves classification consistency across taxonomic levels with minimal additional parameters.

Contribution

It proposes a novel framework combining Tree-Path KL Divergence and Hierarchy-Sibling Smoothed Cross-Entropy for efficient, structure-aware adaptation of VLMs.

Findings

Improves Full-Path Accuracy across benchmarks.

Reduces Tree-based Inconsistency Error.

Requires minimal parameter overhead.

Abstract

Vision-Language Models (VLMs) learn powerful multimodal representations through large-scale image-text pretraining, but adapting them to hierarchical classification is underexplored. Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions across taxonomy levels. We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency. We combine two objectives: Tree-Path KL Divergence (TP-KL) aligns predictions along the ground-truth label path for vertical coherence, while Hierarchy-Sibling Smoothed Cross-Entropy (HiSCE) encourages consistent predictions among sibling classes. Both losses work in the VLM's shared embedding space and integrate with lightweight LoRA adaptation. Experiments across multiple benchmarks show consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error with minimal parameter overhead. Our approach provides an efficient strategy for adapting VLMs to structured taxonomies.