Renal digital pathology visual knowledge search platform based on language large model and book knowledge

TL;DR

This paper presents a renal digital pathology search platform that leverages large language and image models to enable semantic retrieval of pathology images based on text descriptions, enhancing diagnostic support.

Contribution

It introduces a novel knowledge base of over 10,000 renal pathology images with paired descriptions and evaluates multiple large models for semantic and image feature retrieval capabilities.

Findings

Built a comprehensive renal pathology image-text knowledge base

Evaluated the semantic understanding of four large language models

Developed a semantic image retrieval system for renal pathology

Abstract

Large models have become mainstream, yet their applications in digital pathology still require exploration. Meanwhile renal pathology images play an important role in the diagnosis of renal diseases. We conducted image segmentation and paired corresponding text descriptions based on 60 books for renal pathology, clustering analysis for all image and text description features based on large models, ultimately building a retrieval system based on the semantic features of large models. Based above analysis, we established a knowledge base of 10,317 renal pathology images and paired corresponding text descriptions, and then we evaluated the semantic feature capabilities of 4 large models, including GPT2, gemma, LLma and Qwen, and the image-based feature capabilities of dinov2 large model. Furthermore, we built a semantic retrieval system to retrieve pathological images based on text descriptions, and named RppD (aidp.zjsru.edu.cn).