PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms

TL;DR

PanORama introduces a multiview-consistent panoptic segmentation method for operating rooms, improving spatial understanding without camera calibration and outperforming previous methods on key datasets.

Contribution

It is the first to achieve multiview-consistent panoptic segmentation in ORs, modeling cross-view interactions within the backbone in a single pass.

Findings

Achieves over 70% Panoptic Quality on MM-OR and 4D-OR datasets.

Outperforms previous state-of-the-art methods.

Calibration-free and generalizes to unseen viewpoints.

Abstract

Operating rooms (ORs) are cluttered, dynamic, highly occluded environments, where reliable spatial understanding is essential for situational awareness during complex surgical workflows. Achieving spatial understanding for panoptic segmentation from sparse multiview images poses a fundamental challenge, as limited visibility in a subset of views often leads to mispredictions across cameras. To this end, we introduce PanORama, the first panoptic segmentation for the operating room that is multiview-consistent by design. By modeling cross-view interactions at the feature level inside the backbone in a single forward pass, view consistency emerges directly rather than through post-hoc refinement. We evaluate on the MM-OR and 4D-OR datasets, achieving >70% Panoptic Quality (PQ) performance, and outperforming the previous state of the art. Importantly, PanORama is calibration-free, requiring no camera parameters, and generalizes to unseen camera viewpoints within any multiview configuration at inference time. By substantially enhancing multiview segmentation and, consequently, spatial understanding in the OR, we believe our approach opens new opportunities for surgical perception and assistance. Code will be released upon acceptance.