Efficient information extraction using LLMs and knowledge distillation: A study on HPV health communication

TL;DR

This study uses efficient AI models to evaluate how state health websites communicate HPV and vaccination information, aiming to improve public health messaging.

Contribution

A computationally efficient framework using knowledge distillation to evaluate health communication on HPV from state websites.

Findings

A fine-tuned RoBERTa Large model achieved an F1 score of 0.74, nearly matching the teacher model's performance.

The framework was applied to evaluate content from 48 state health department websites.

The method can identify gaps in health communication and improve messaging for public health.

Abstract

State Department of Health (DOH) websites serve as authoritative sources of HPV-related health communications, presenting state-specific content that influences public awareness and vaccination decisions. We develop a computationally efficient framework to systematically evaluate these information repositories based on their content quality, completeness, and their motivational impact on vaccination behavior. We propose a dataset consolidating 48 different DOH websites’ data targeted towards HPV and HPV vaccination. By developing an annotated dataset (n = 400), efficient prompting techniques and a Knowledge Distillation framework, we develop and evaluate efficient student models based on the Llama family of Large Language Models (LLMs) and the RoBERTa Large encoder architecture. We finally deploy the best-performing student model for a computationally feasible evaluation of the content of DOH websites. We show that fine-tuned RoBERTa Large model achieves an F1 score of 0.74 on the test set, outperforming all other student models and approaching the teacher model's performance (F1 = 0.77). The fine-tuned RoBERTa-Large model is subsequently applied to data from various state DOH websites to evaluate the information presented. We also discuss the broader implications, limitations, and ethical and legal considerations of the proposed approach.

We studied how state health department websites communicate information about human papillomavirus (HPV) and vaccination. These websites are important because they shape public understanding and influence people’s decisions about getting vaccinated. We collected information from 48 state websites and looked at how complete, clear, and persuasive their content was. To do this, we created a smaller, efficient computer model that could quickly read and evaluate large amounts of website text. We trained this model using a teaching approach where a larger, more powerful model showed it how to make good decisions. Our smaller model was almost as accurate as the larger one, but much faster and easier to use. We then used it to review each state’s website. This approach can help public health organizations identify gaps in the information they provide and improve how they communicate important health messages. While we focused on HPV, the same method could be used to study how other health topics are presented online.

Figures