Health-SCORE: Towards Scalable Rubrics for Improving Health-LLMs

TL;DR

Health-SCORE introduces a scalable rubric-based framework for evaluating and training healthcare-related language models, reducing development costs while maintaining high evaluation quality and enabling safety-aware reinforcement learning.

Contribution

It presents a generalizable, cost-effective rubric framework that enhances model evaluation and training in healthcare without sacrificing performance.

Findings

Health-SCORE matches human rubric evaluation quality.

It reduces rubric development effort significantly.

It enables safety-aware reinforcement learning and improved in-context learning.

Abstract

Rubrics are essential for evaluating open-ended LLM responses, especially in safety-critical domains such as healthcare. However, creating high-quality and domain-specific rubrics typically requires significant human expertise time and development cost, making rubric-based evaluation and training difficult to scale. In this work, we introduce Health-SCORE, a generalizable and scalable rubric-based training and evaluation framework that substantially reduces rubric development costs without sacrificing performance. We show that Health-SCORE provides two practical benefits beyond standalone evaluation: it can be used as a structured reward signal to guide reinforcement learning with safety-aware supervision, and it can be incorporated directly into prompts to improve response quality through in-context learning. Across open-ended healthcare tasks, Health-SCORE achieves evaluation quality comparable to human-created rubrics while significantly lowering development effort, making rubric-based evaluation and training more scalable.