Visually Consistent Hierarchical Image Classification

TL;DR

This paper introduces a novel approach for hierarchical image classification that enforces internal visual consistency via intra-image segmentation, leading to improved accuracy and more reliable predictions without needing pixel-level annotations.

Contribution

It is the first method to align fine-to-coarse predictions through intra-image segmentation to ensure visual consistency in hierarchical classification.

Findings

Outperforms zero-shot CLIP and state-of-the-art baselines

Achieves higher accuracy in hierarchical classification

Improves internal image segmentation without pixel annotations

Abstract

Hierarchical classification predicts labels across multiple levels of a taxonomy, e.g., from coarse-level 'Bird' to mid-level 'Hummingbird' to fine-level 'Green hermit', allowing flexible recognition under varying visual conditions. It is commonly framed as multiple single-level tasks, but each level may rely on different visual cues: Distinguishing 'Bird' from 'Plant' relies on global features like feathers or leaves, while separating 'Anna's hummingbird' from 'Green hermit' requires local details such as head coloration. Prior methods improve accuracy using external semantic supervision, but such statistical learning criteria fail to ensure consistent visual grounding at test time, resulting in incorrect hierarchical classification. We propose, for the first time, to enforce internal visual consistency by aligning fine-to-coarse predictions through intra-image segmentation. Our method outperforms zero-shot CLIP and state-of-the-art baselines on hierarchical classification benchmarks, achieving both higher accuracy and more consistent predictions. It also improves internal image segmentation without requiring pixel-level annotations.