Microscope 2.0: An Augmented Reality Microscope with Real-time Artificial Intelligence Integration

TL;DR

Microscope 2.0 introduces an augmented reality system that overlays real-time AI analysis onto microscope views, enhancing cancer diagnosis accuracy and workflow efficiency without requiring costly digitization or extensive AI deployment.

Contribution

The paper presents a cost-effective augmented reality microscope that seamlessly integrates real-time AI analysis into traditional microscopy workflows for improved cancer diagnosis.

Findings

Effective detection of lymph node metastases in breast cancer

Identification of prostate cancer with real-time latency

Potential to improve diagnostic accuracy and workflow efficiency

Abstract

The brightfield microscope is instrumental in the visual examination of both biological and physical samples at sub-millimeter scales. One key clinical application has been in cancer histopathology, where the microscopic assessment of the tissue samples is used for the diagnosis and staging of cancer and thus guides clinical therapy. However, the interpretation of these samples is inherently subjective, resulting in significant diagnostic variability. Moreover, in many regions of the world, access to pathologists is severely limited due to lack of trained personnel. In this regard, Artificial Intelligence (AI) based tools promise to improve the access and quality of healthcare. However, despite significant advances in AI research, integration of these tools into real-world cancer diagnosis workflows remains challenging because of the costs of image digitization and difficulties in deploying AI solutions. Here we propose a cost-effective solution to the integration of AI: the Augmented Reality Microscope (ARM). The ARM overlays AI-based information onto the current view of the sample through the optical pathway in real-time, enabling seamless integration of AI into the regular microscopy workflow. We demonstrate the utility of ARM in the detection of lymph node metastases in breast cancer and the identification of prostate cancer with a latency that supports real-time workflows. We anticipate that ARM will remove barriers towards the use of AI in microscopic analysis and thus improve the accuracy and efficiency of cancer diagnosis. This approach is applicable to other microscopy tasks and AI algorithms in the life sciences and beyond.