ManiNeg: Manifestation-guided Multimodal Pretraining for Mammography Classification

TL;DR

ManiNeg introduces a manifestation-guided approach for hard negative sampling in contrastive learning, enhancing mammography classification accuracy and generalization, supported by a new multi-view dataset with detailed annotations.

Contribution

This work presents ManiNeg, a novel manifestation-based hard negative sampling method for contrastive learning in mammography, along with the MVKL dataset for improved breast cancer classification.

Findings

Improved classification accuracy with ManiNeg

Enhanced representation learning in unimodal and multimodal settings

Demonstrated generalization across datasets

Abstract

Breast cancer is a significant threat to human health. Contrastive learning has emerged as an effective method to extract critical lesion features from mammograms, thereby offering a potent tool for breast cancer screening and analysis. A crucial aspect of contrastive learning involves negative sampling, where the selection of appropriate hard negative samples is essential for driving representations to retain detailed information about lesions. In contrastive learning, it is often assumed that features can sufficiently capture semantic content, and that each minibatch inherently includes ideal hard negative samples. However, the characteristics of breast lumps challenge these assumptions. In response, we introduce ManiNeg, a novel approach that leverages manifestations as proxies to mine hard negative samples. Manifestations, which refer to the observable symptoms or signs of a disease, provide a knowledge-driven and robust basis for choosing hard negative samples. This approach benefits from its invariance to model optimization, facilitating efficient sampling. To support ManiNeg and future research endeavors, we developed the MVKL dataset, which includes multi-view mammograms, corresponding reports, meticulously annotated manifestations, and pathologically confirmed benign-malignant outcomes. We evaluate ManiNeg on the benign and malignant classification task. Our results demonstrate that ManiNeg not only improves representation in both unimodal and multimodal contexts but also shows generalization across datasets. The MVKL dataset and our codes are publicly available at https://github.com/wxwxwwxxx/ManiNeg.