Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation

TL;DR

Hyp2Former introduces a hierarchy-aware hyperbolic embedding framework for open-set panoptic segmentation, effectively distinguishing known and unknown objects by leveraging semantic hierarchies without explicit unknown modeling during training.

Contribution

It proposes a novel end-to-end method that encodes hierarchical semantic relationships in hyperbolic space, improving unknown object detection in OPS tasks.

Findings

Outperforms existing OPS methods on multiple datasets.

Effectively detects unknown objects by proximity to higher-level concepts.

Achieves a good balance between discovering unknowns and maintaining robustness.

Abstract

Recognizing unknown objects is crucial for safety-critical applications such as autonomous driving and robotics. Open-Set Panoptic Segmentation (OPS) aims to segment known thing and stuff classes while identifying valid unknown objects as separate instances. Prior OPS approaches largely treat known categories as a flat label set, ignoring the semantic hierarchy that provides valuable structural priors for distinguishing unknown objects from in-distribution classes. In this work, we propose Hyp2Former, an end-to-end framework for OPS that does not require explicit modeling of unknowns during training, and instead learns hierarchical semantic similarities continuously in hyperbolic space. By explicitly encoding hierarchical relationships among known categories, the model learns a structured embedding space that captures multiple levels of semantic abstraction. As a result, unknown objects that cannot be confidently classified as known categories still remain in close proximity to higher-level concepts (e.g., an unknown animal remains closer to "animal" or "object" than to unrelated concepts such as "electronics" or "stuff") and can therefore be reliably detected, even if their fine-grained category was not represented during training. Empirical evaluations across multiple public datasets such as MS COCO, Cityscapes, and Lost&Found demonstrate that Hyp2Former outperforms existing methods on OPS, achieving the best balance between unknown object discovery and in-distribution robustness.