A Novel Approach for Pill-Prescription Matching with GNN Assistance and Contrastive Learning

TL;DR

This paper introduces PIMA, a novel GNN and contrastive learning-based system for accurately matching pill images with prescription text to reduce medication errors.

Contribution

The paper presents a new approach combining GNN and contrastive learning for pill-prescription matching, significantly improving accuracy over baseline models.

Findings

PIMA achieves 46.95% accuracy, outperforming baselines.

GNN effectively models spatial correlations in prescriptions.

Contrastive learning enhances cross-modal similarity understanding.

Abstract

Medication mistaking is one of the risks that can result in unpredictable consequences for patients. To mitigate this risk, we develop an automatic system that correctly identifies pill-prescription from mobile images. Specifically, we define a so-called pill-prescription matching task, which attempts to match the images of the pills taken with the pills' names in the prescription. We then propose PIMA, a novel approach using Graph Neural Network (GNN) and contrastive learning to address the targeted problem. In particular, GNN is used to learn the spatial correlation between the text boxes in the prescription and thereby highlight the text boxes carrying the pill names. In addition, contrastive learning is employed to facilitate the modeling of cross-modal similarity between textual representations of pill names and visual representations of pill images. We conducted extensive experiments and demonstrated that PIMA outperforms baseline models on a real-world dataset of pill and prescription images that we constructed. Specifically, PIMA improves the accuracy from 19.09% to 46.95% compared to other baselines. We believe our work can open up new opportunities to build new clinical applications and improve medication safety and patient care.