Identifying Evidence-Based Nudges in Biomedical Literature with Large Language Models

TL;DR

This paper introduces an AI system that efficiently identifies evidence-based behavioral nudges from vast biomedical literature using a multi-stage filtering and classification pipeline, aiding health intervention discovery.

Contribution

The work presents a novel multi-stage pipeline combining hybrid filtering and LLM classification to extract structured evidence-based nudges from over 8 million biomedical articles.

Findings

Achieved 67% F1 score and 72% recall in identifying nudges.

High-precision variant reached 100% precision with 12% recall.

System is integrated into a real-world healthcare platform.

Abstract

We present a scalable, AI-powered system that identifies and extracts evidence-based behavioral nudges from unstructured biomedical literature. Nudges are subtle, non-coercive interventions that influence behavior without limiting choice, showing strong impact on health outcomes like medication adherence. However, identifying these interventions from PubMed's 8 million+ articles is a bottleneck. Our system uses a novel multi-stage pipeline: first, hybrid filtering (keywords, TF-IDF, cosine similarity, and a "nudge-term bonus") reduces the corpus to about 81,000 candidates. Second, we use OpenScholar (quantized LLaMA 3.1 8B) to classify papers and extract structured fields like nudge type and target behavior in a single pass, validated against a JSON schema. We evaluated four configurations on a labeled test set (N=197). The best setup (Title/Abstract/Intro) achieved a 67.0% F1 score and 72.0% recall, ideal for discovery. A high-precision variant using self-consistency (7 randomized passes) achieved 100% precision with 12% recall, demonstrating a tunable trade-off for high-trust use cases. This system is being integrated into Agile Nudge+, a real-world platform, to ground LLM-generated interventions in peer-reviewed evidence. This work demonstrates interpretable, domain-specific retrieval pipelines for evidence synthesis and personalized healthcare.