Covariance Descriptors Meet General Vision Encoders: Riemannian Deep Learning for Medical Image Classification

TL;DR

This paper explores the use of covariance descriptors derived from pre-trained general vision encoders, combined with SPDNet, to improve medical image classification performance, demonstrating their superiority over handcrafted features across multiple datasets.

Contribution

It introduces a novel approach of constructing covariance descriptors from GVE features and evaluates their effectiveness with SPDNet in medical imaging, showing significant performance gains.

Findings

Covariance descriptors from GVE features outperform handcrafted descriptors.

SPDNet with DINOv2 features achieves state-of-the-art results.

The approach is effective across multiple medical imaging datasets.

Abstract

Covariance descriptors capture second-order statistics of image features. They have shown strong performance in general computer vision tasks, but remain underexplored in medical imaging. We investigate their effectiveness for both conventional and learning-based medical image classification, with a particular focus on SPDNet, a classification network specifically designed for symmetric positive definite (SPD) matrices. We propose constructing covariance descriptors from features extracted by pre-trained general vision encoders (GVEs) and comparing them with handcrafted descriptors. Two GVEs - DINOv2 and MedSAM - are evaluated across eleven binary and multi-class datasets from the MedMNSIT benchmark. Our results show that covariance descriptors derived from GVE features consistently outperform those derived from handcrafted features. Moreover, SPDNet yields superior performance to state-of-the-art methods when combined with DINOv2 features. Our findings highlight the potential of combining covariance descriptors with powerful pretrained vision encoders for medical image analysis.