Beyond the Monitor: Mixed Reality Visualization and Multimodal AI for Enhanced Digital Pathology Workflow

TL;DR

PathVis is a mixed-reality platform for Apple Vision Pro that enhances digital pathology workflows by integrating immersive visualization with multimodal AI, reducing cognitive load and streamlining diagnosis of gigapixel whole-slide images.

Contribution

The paper introduces PathVis, a novel mixed-reality system that combines immersive visualization and multimodal AI to improve digital pathology workflows.

Findings

Feasibility demonstrated through preliminary qualitative evaluation

Enhanced navigation with embodied interaction methods

Integration of AI for case retrieval and interpretation

Abstract

Pathologists diagnose cancer using gigapixel whole-slide images (WSIs), but the current digital workflow is fragmented. These multiscale datasets often exceed 100,000 x 100,000 pixels, yet standard 2D monitors restrict the field of view. This disparity forces constant panning and zooming, which increases cognitive load and disrupts diagnostic momentum. We introduce PathVis, a mixed-reality platform for Apple Vision Pro that unifies this ecosystem into a single immersive environment. PathVis replaces indirect mouse navigation with embodied interaction, utilizing eye gaze, natural hand gestures, and voice commands to explore gigapixel data. The system integrates multimodal AI agents to support computer-aided diagnosis: a content-based image retrieval engine spatially displays similar patient cases for side-by-side prognostic comparison, while a conversational assistant provides real-time interpretation. By merging immersive visualization with integrated AI capabilities, PathVis shows promise in streamlining diagnostic workflows and mitigating the burden of context switching. This paper presents the system architecture and a preliminary qualitative evaluation demonstrating the platform's feasibility. The PathVis source code and a demo video are publicly available at: https://github.com/jaiprakash1824/Path_Vis.