Mining Healthcare Procurement Data Using Text Mining and Natural Language Processing -- Reflection From An Industrial Project

TL;DR

This paper presents a real-world application of text mining and NLP techniques to extract structured data from heterogeneous, multilingual healthcare procurement documents, enabling better supplier risk assessment and procurement processes.

Contribution

It introduces a novel approach to handle data heterogeneity and multilingual challenges in healthcare procurement documents, resulting in the creation of the first structured procurement contract database.

Findings

Successfully extracted structured data from millions of documents.

Developed a method that generalizes across multiple languages and data types.

Provided practical insights and recommendations for deploying NLP in industry.

Abstract

While text mining and NLP research has been established for decades, there remain gaps in the literature that reports the use of these techniques in building real-world applications. For example, they typically look at single and sometimes simplified tasks, and do not discuss in-depth data heterogeneity and inconsistency that is common in real-world problems or their implication on the development of their methods. Also, few prior work has focused on the healthcare domain. In this work, we describe an industry project that developed text mining and NLP solutions to mine millions of heterogeneous, multilingual procurement documents in the healthcare sector. We extract structured procurement contract data that is used to power a platform for dynamically assessing supplier risks. Our work makes unique contributions in a number of ways. First, we deal with highly heterogeneous, multilingual data and we document our approach to tackle these challenges. This is mainly based on a method that effectively uses domain knowledge and generalises to multiple text mining and NLP tasks and languages. Second, applying this method to mine millions of procurement documents, we develop the first structured procurement contract database that will help facilitate the tendering process. Second, Finally, we discuss lessons learned for practical text mining/NLP development, and make recommendations for future research and practice.