Quantify unmet medical need across the disease landscape – A large language model-based methodology

TL;DR

This paper introduces a scalable AI method to quantify unmet medical need across all human diseases, enabling faster and cheaper prioritization of research efforts.

Contribution

A novel LLM-based framework to systematically score unmet medical need across 22,701 diseases using 11 criteria, enabling rapid and low-cost prioritization.

Findings

The LLM-based system scored all 22,701 diseases in ~1 hour at a cost of $120 USD.

LLM scores showed strong agreement with expert rankings (e.g., Spearman’s rho of 0.845 for mortality).

The method is flexible, allowing organizations to adjust criteria weights based on their priorities.

Abstract

Despite the ultimate goal of medical researchers and funders being to maximize patient benefit, there is no systematic process for quantifying unmet medical need across diseases. While a relative unmet medical need scoring system would be valuable for prioritization of medical research, systematically performing this effort across all 22,701 human diseases is technically challenging, time-consuming, and expensive. Using a large language model-based (LLM) architecture, we built a scalable method demonstrating feasibility to quantify “unmet medical need” criteria across all diseases, combine those criteria into a single weighted score, and extend the method into new criteria or diseases in the future. We aimed to quantitatively determine which diseases have the greatest unmet medical need and, therefore, which diseases are priority targets for new repurposed treatments.

We defined 11 scoring criteria across three categories of unmet medical need. For each criterion, we tested LLM models and refined prompts to generate a score per criteria for each disease and then defined a weighting for each criterion to contribute to a final score. A 30-disease development set was used to iterate on the prompting, and a 10-disease evaluation set was held out and used to evaluate the performance of the final prompt. All 22,701 human diseases in the MONDO disease ontology were quantitatively scored for their unmet medical need across all 11 weighted criteria. The resulting scores allowed for relative comparison between diseases of unmet medical needs. Inter-expert agreement was strong, indicating reliability of the scoring framework with 95% of ratings within a 1-point difference. Across multiple LLMs, gpt-4o is most closely aligned with expert rankings, achieving low mean and standard deviation differences relative to human scores. Furthermore, LLM-generated scores demonstrated strong Spearman’s rho correlations with expert assessments across key clinical criteria, such as mortality (ρ = 0.845) and quality-adjusted life years lost (ρ = 0.822), supporting their suitability for prioritizing unmet medical need. All data were generated in ~1 hour with no missing data, at a total cost of $120 USD of compute and the results of the Unmet Medical Need Index are publicly available. The main limitation of this study is the combined size of the development and evaluation set being 40 diseases.

This accessible, scalable methodology enables funders and researchers, across governments, universities, healthcare organizations, and disease groups to tailor prioritization efforts according to unmet medical need in the context of their organizational objectives, by selecting appropriate criteria and weighting of those criteria. This method creates a pragmatic and transparent tool to streamline research prioritization. Future research should consider expanding the disease set size used to create scores.

We identified that there is currently no global, systematic database or technique that measures and ranks the “unmet medical need” across all known human diseases.

Relative scores of unmet medical need would help with prioritizing research activities across diseases, but manually assessing all diseases is impractical because it is too expensive and requires years of specialized expertise.

A scalable method is needed to help organizations prioritize their funding and research toward the diseases where new treatments will have the greatest impact on saving and improving patient lives.

We developed an artificial intelligence method that evaluates 11 different factors across three areas (patient suffering, standard of care, and accessibility) to create an “unmet medical need score” for every known disease.

We found that this system could accurately score 22,701 diseases in ~1 hour for a total computing cost of only $120.

Our analysis showed that the scores generated by artificial intelligence closely matched the rankings provided by a panel of medical experts.

We have provided an accessible and low-cost method that allows governments, universities, and nonprofits/charities to more objectively understand the relative unmet medical need across diseases, which could be used to prioritize diseases in need of research attention.

This method is highly flexible, allowing different organizations to adjust the importance of specific factors, such as prioritizing rare childhood diseases or those with the highest death rates, to match their specific mission.

While this system is a powerful guide for comparing thousands of conditions, we recommend using the scores in combination with other methods rather than as a final, standalone judgment.

The main limitation of this study is the combined size of the development and evaluation set being 40 diseases, which is helpful to be able to develop scores quickly, but means the ability to generalize these results is limited.

Elliott W Sharp and colleagues develop Large Language Model-Based Methodology to quantify unmet needs among human diseases considering factors related to patient suffering, standard of care, and accessibility of care.

Figures