A Spatial-Temporal Deformable Attention based Framework for Breast Lesion Detection in Videos

TL;DR

This paper introduces STNet, a novel spatial-temporal deformable attention framework for breast lesion detection in videos, improving deep feature aggregation and detection speed over existing methods.

Contribution

The paper proposes a spatial-temporal deformable attention module and an encoder feature shuffle strategy, enabling effective local feature fusion and faster multi-frame detection.

Findings

Achieves state-of-the-art detection performance on breast ultrasound videos.

Operates twice as fast in inference compared to previous methods.

Effectively fuses local spatial-temporal features for improved accuracy.

Abstract

Detecting breast lesion in videos is crucial for computer-aided diagnosis. Existing video-based breast lesion detection approaches typically perform temporal feature aggregation of deep backbone features based on the self-attention operation. We argue that such a strategy struggles to effectively perform deep feature aggregation and ignores the useful local information. To tackle these issues, we propose a spatial-temporal deformable attention based framework, named STNet. Our STNet introduces a spatial-temporal deformable attention module to perform local spatial-temporal feature fusion. The spatial-temporal deformable attention module enables deep feature aggregation in each stage of both encoder and decoder. To further accelerate the detection speed, we introduce an encoder feature shuffle strategy for multi-frame prediction during inference. In our encoder feature shuffle strategy, we share the backbone and encoder features, and shuffle encoder features for decoder to generate the predictions of multiple frames. The experiments on the public breast lesion ultrasound video dataset show that our STNet obtains a state-of-the-art detection performance, while operating twice as fast inference speed. The code and model are available at https://github.com/AlfredQin/STNet.