U-LanD: Uncertainty-Driven Video Landmark Detection

TL;DR

U-LanD is a novel framework that leverages predictive uncertainty in Bayesian models to automatically identify key frames in videos for landmark detection, effectively handling noisy and sparse labels.

Contribution

It introduces an uncertainty-driven approach to automatically recognize key frames for landmark detection in videos with noisy, sparse labels, demonstrated on ultrasound heart videos.

Findings

U-LanD outperforms non-Bayesian methods by 42% in R2 score.

The approach requires minimal additional computational overhead.

It is applicable to other challenging datasets with noisy labels.

Abstract

This paper presents U-LanD, a framework for joint detection of key frames and landmarks in videos. We tackle a specifically challenging problem, where training labels are noisy and highly sparse. U-LanD builds upon a pivotal observation: a deep Bayesian landmark detector solely trained on key video frames, has significantly lower predictive uncertainty on those frames vs. other frames in videos. We use this observation as an unsupervised signal to automatically recognize key frames on which we detect landmarks. As a test-bed for our framework, we use ultrasound imaging videos of the heart, where sparse and noisy clinical labels are only available for a single frame in each video. Using data from 4,493 patients, we demonstrate that U-LanD can exceedingly outperform the state-of-the-art non-Bayesian counterpart by a noticeable absolute margin of 42% in R2 score, with almost no overhead imposed on the model size. Our approach is generic and can be potentially applied to other challenging data with noisy and sparse training labels.