manvr3d: A Platform for Human-in-the-loop Cell Tracking in Virtual Reality

TL;DR

manvr3d introduces a VR platform that integrates human-in-the-loop cell tracking with 3D visualization, enabling more intuitive and efficient annotation, proofreading, and analysis of high-resolution microscopy data for life scientists.

Contribution

This work presents a novel VR-enabled platform that combines deep learning-based cell tracking with 3D visualization and natural user interfaces for improved workflow efficiency.

Findings

Enhanced annotation speed with VR controllers and eye-tracking

Improved spatial understanding through 3D visualization

Facilitated rapid ground truth generation and proofreading

Abstract

We propose manvr3d, a novel VR-ready platform for interactive human-in-the-loop cell tracking. We utilize VR controllers and eye-tracking hardware to facilitate rapid ground truth generation and proofreading for deep learning-based cell tracking models. Life scientists reconstruct the developmental history of organisms on the cellular level by analyzing 3D time-lapse microscopy images acquired at high spatio-temporal resolution. The reconstruction of such cell lineage trees traditionally involves tracking individual cells through all recorded time points, manually annotating their positions, and then linking them over time to create complete trajectories. Deep learning-based algorithms accelerate this process, yet depend heavily on manually-annotated high-quality ground truth data and curation. Visual representation of the image data in this process still relies primarily on 2D renderings, which greatly limits spatial understanding and navigation. In this work, we bridge the gap between deep learning-based cell tracking software and 3D/VR visualization to create a human-in-the-loop cell tracking system. We lift the incremental annotation, training and proofreading loop of the deep learning model into the 3rd dimension and apply natural user interfaces like hand gestures and eye tracking to accelerate the cell tracking workflow for life scientists.