Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19

TL;DR

This paper presents a novel approach using fine-tuned generative large language models in a conversational AI app for personalized COVID-19 risk assessment, outperforming traditional classifiers in low-data settings.

Contribution

It introduces a no-code, interactive risk assessment system leveraging streaming QA and attention-based interpretability, demonstrating effectiveness with limited training data.

Findings

High AUC scores with few fine-tuning samples

Outperforms traditional classifiers in low-data regimes

Enables real-time, personalized risk assessment via conversational AI

Abstract

Large language models (LLMs) have shown remarkable capabilities in various natural language tasks and are increasingly being applied in healthcare domains. This work demonstrates a new LLM-powered disease risk assessment approach via streaming human-AI conversation, eliminating the need for programming required by traditional machine learning approaches. In a COVID-19 severity risk assessment case study, we fine-tune pre-trained generative LLMs (e.g., Llama2-7b and Flan-t5-xl) using a few shots of natural language examples, comparing their performance with traditional classifiers (i.e., Logistic Regression, XGBoost, Random Forest) that are trained de novo using tabular data across various experimental settings. We develop a mobile application that uses these fine-tuned LLMs as its generative AI (GenAI) core to facilitate real-time interaction between clinicians and patients, providing no-code risk assessment through conversational interfaces. This integration not only allows for the use of streaming Questions and Answers (QA) as inputs but also offers personalized feature importance analysis derived from the LLM's attention layers, enhancing the interpretability of risk assessments. By achieving high Area Under the Curve (AUC) scores with a limited number of fine-tuning samples, our results demonstrate the potential of generative LLMs to outperform discriminative classification methods in low-data regimes, highlighting their real-world adaptability and effectiveness. This work aims to fill the existing gap in leveraging generative LLMs for interactive no-code risk assessment and to encourage further research in this emerging field.