Semantic Scene Graph for Ultrasound Image Explanation and Scanning Guidance

TL;DR

This paper introduces a scene graph approach for ultrasound images that explains content and guides scanning, aiming to improve interpretability and usability for non-expert users in point-of-care settings.

Contribution

The study presents a transformer-based scene graph method for ultrasound images combined with LLMs to provide explanations and scanning guidance, enhancing accessibility for non-experts.

Findings

Effective scene graph generation without explicit object detection

Improved ultrasound interpretability for non-experts

Potential to standardize and complete anatomical exploration

Abstract

Understanding medical ultrasound imaging remains a long-standing challenge due to significant visual variability caused by differences in imaging and acquisition parameters. Recent advancements in large language models (LLMs) have been used to automatically generate terminology-rich summaries orientated to clinicians with sufficient physiological knowledge. Nevertheless, the increasing demand for improved ultrasound interpretability and basic scanning guidance among non-expert users, e.g., in point-of-care settings, has not yet been explored. In this study, we first introduce the scene graph (SG) for ultrasound images to explain image content to ordinary and provide guidance for ultrasound scanning. The ultrasound SG is first computed using a transformer-based one-stage method, eliminating the need for explicit object detection. To generate a graspable image explanation for ordinary, the user query is then used to further refine the abstract SG representation through LLMs. Additionally, the predicted SG is explored for its potential in guiding ultrasound scanning toward missing anatomies within the current imaging view, assisting ordinary users in achieving more standardized and complete anatomical exploration. The effectiveness of this SG-based image explanation and scanning guidance has been validated on images from the left and right neck regions, including the carotid and thyroid, across five volunteers. The results demonstrate the potential of the method to maximally democratize ultrasound by enhancing its interpretability and usability for ordinaries.