Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilities

TL;DR

Syn-Mediverse is a large, hyper-realistic synthetic multimodal dataset designed to improve scene understanding in healthcare facilities, supporting various computer vision tasks for autonomous healthcare robots.

Contribution

It introduces the first comprehensive synthetic dataset for healthcare environments, enabling training and benchmarking of models across multiple scene understanding tasks.

Findings

Dataset contains over 48,000 images and 1.5 million annotations.

Evaluations show the dataset's complexity and challenge for state-of-the-art models.

Provides an online benchmark for further research.

Abstract

Safety and efficiency are paramount in healthcare facilities where the lives of patients are at stake. Despite the adoption of robots to assist medical staff in challenging tasks such as complex surgeries, human expertise is still indispensable. The next generation of autonomous healthcare robots hinges on their capacity to perceive and understand their complex and frenetic environments. While deep learning models are increasingly used for this purpose, they require extensive annotated training data which is impractical to obtain in real-world healthcare settings. To bridge this gap, we present Syn-Mediverse, the first hyper-realistic multimodal synthetic dataset of diverse healthcare facilities. Syn-Mediverse contains over \num{48000} images from a simulated industry-standard optical tracking camera and provides more than 1.5M annotations spanning five different scene understanding tasks including depth estimation, object detection, semantic segmentation, instance segmentation, and panoptic segmentation. We demonstrate the complexity of our dataset by evaluating the performance on a broad range of state-of-the-art baselines for each task. To further advance research on scene understanding of healthcare facilities, along with the public dataset we provide an online evaluation benchmark available at \url{http://syn-mediverse.cs.uni-freiburg.de}