Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data

TL;DR

This paper introduces RASO, a foundation model for recognizing any surgical object using weakly-supervised learning from large-scale unannotated videos, achieving significant improvements in open-set recognition and surgical action recognition.

Contribution

The paper presents a novel weakly-supervised learning framework and a scalable data generation pipeline for surgical object recognition, reducing manual annotation needs and enhancing recognition performance.

Findings

RASO improves zero-shot recognition accuracy by up to 10.6 mAP.

The data pipeline produces 3.6 million annotations from 2,200 procedures.

RASO surpasses state-of-the-art models in supervised surgical action recognition.

Abstract

We present RASO, a foundation model designed to Recognize Any Surgical Object, offering robust open-set recognition capabilities across a broad range of surgical procedures and object classes, in both surgical images and videos. RASO leverages a novel weakly-supervised learning framework that generates tag-image-text pairs automatically from large-scale unannotated surgical lecture videos, significantly reducing the need for manual annotations. Our scalable data generation pipeline gathers 2,200 surgical procedures and produces 3.6 million tag annotations across 2,066 unique surgical tags. Our experiments show that RASO achieves improvements of 2.9 mAP, 4.5 mAP, 10.6 mAP, and 7.2 mAP on four standard surgical benchmarks, respectively, in zero-shot settings, and surpasses state-of-the-art models in supervised surgical action recognition tasks. Code, model, and demo are available at https://ntlm1686.github.io/raso.