Medicinal Boxes Recognition on a Deep Transfer Learning Augmented Reality Mobile Application

TL;DR

This paper presents a mobile augmented reality app that uses deep transfer learning with DenseNet to recognize medicines and display relevant information, aiding patients in correct medication intake.

Contribution

It introduces a novel AR application combining deep transfer learning for medicine recognition with real-time information overlay, improving patient medication adherence.

Findings

Achieved up to 91.30% recognition accuracy.

Demonstrated real-time performance on mobile devices.

Optimized hyperparameters for best recognition results.

Abstract

Taking medicines is a fundamental aspect to cure illnesses. However, studies have shown that it can be hard for patients to remember the correct posology. More aggravating, a wrong dosage generally causes the disease to worsen. Although, all relevant instructions for a medicine are summarized in the corresponding patient information leaflet, the latter is generally difficult to navigate and understand. To address this problem and help patients with their medication, in this paper we introduce an augmented reality mobile application that can present to the user important details on the framed medicine. In particular, the app implements an inference engine based on a deep neural network, i.e., a densenet, fine-tuned to recognize a medicinal from its package. Subsequently, relevant information, such as posology or a simplified leaflet, is overlaid on the camera feed to help a patient when taking a medicine. Extensive experiments to select the best hyperparameters were performed on a dataset specifically collected to address this task; ultimately obtaining up to 91.30\% accuracy as well as real-time capabilities.