Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models

TL;DR

This paper introduces TARA, a method that improves hierarchical visual recognition in large multimodal models by aligning visual features with biological taxonomy knowledge, enhancing accuracy for known and novel categories.

Contribution

TARA is a novel approach that injects taxonomic knowledge into LMMs using hierarchical contrastive learning with biological foundation models, improving hierarchical consistency and recognition accuracy.

Findings

Enhances LMMs' hierarchical consistency.

Improves leaf node accuracy for known and novel categories.

Effective in complex biological taxonomies.

Abstract

A high-performing, general-purpose visual understanding model should map visual inputs to a taxonomic tree of labels, identify novel categories beyond the training set for which few or no publicly available images exist. Large Multimodal Models (LMMs) have achieved remarkable progress in fine-grained visual recognition (FGVR) for known categories. However, they remain limited in hierarchical visual recognition (HVR) that aims at predicting consistent label paths from coarse to fine categories, especially for novel categories. To tackle these challenges, we propose Taxonomy-Aware Representation Alignment (TARA), a simple yet effective strategy to inject taxonomic knowledge into LMMs. TARA leverages representations from biology foundation models (BFMs) that encode rich biological relationships through hierarchical contrastive learning. By aligning the intermediate representations of visual features with those of BFMs, LMMs are encouraged to extract discriminative visual cues well structured in the taxonomy tree. Additionally, we align the representations of the first answer token with the ground-truth label, flexibly bridging the gap between contextualized visual features and categories of varying granularity according to user intent. Experiments demonstrate that TARA consistently enhances LMMs' hierarchical consistency and leaf node accuracy, enabling reliable recognition of both known and novel categories within complex biological taxonomies. Code is available at https://github.com/PKU-ICST-MIPL/TARA_CVPR2026.