Latent Graph Representations for Critical View of Safety Assessment

TL;DR

This paper introduces a novel approach for critical view of safety assessment in laparoscopic surgery using a disentangled latent scene graph and graph neural networks, reducing annotation costs and improving robustness.

Contribution

It proposes a new method that encodes semantic and visual information into a latent scene graph for CVS prediction, trained with minimal annotation types.

Findings

Outperforms baseline methods with bounding box annotations

Maintains state-of-the-art performance with segmentation masks

Scales effectively across different annotation levels

Abstract

Assessing the critical view of safety in laparoscopic cholecystectomy requires accurate identification and localization of key anatomical structures, reasoning about their geometric relationships to one another, and determining the quality of their exposure. Prior works have approached this task by including semantic segmentation as an intermediate step, using predicted segmentation masks to then predict the CVS. While these methods are effective, they rely on extremely expensive ground-truth segmentation annotations and tend to fail when the predicted segmentation is incorrect, limiting generalization. In this work, we propose a method for CVS prediction wherein we first represent a surgical image using a disentangled latent scene graph, then process this representation using a graph neural network. Our graph representations explicitly encode semantic information - object location, class information, geometric relations - to improve anatomy-driven reasoning, as well as visual features to retain differentiability and thereby provide robustness to semantic errors. Finally, to address annotation cost, we propose to train our method using only bounding box annotations, incorporating an auxiliary image reconstruction objective to learn fine-grained object boundaries. We show that our method not only outperforms several baseline methods when trained with bounding box annotations, but also scales effectively when trained with segmentation masks, maintaining state-of-the-art performance.