A Course Shared Task on Evaluating LLM Output for Clinical Questions

TL;DR

This paper introduces a shared task designed to evaluate how well large language models generate safe and accurate responses to clinical health questions, aiming to improve NLP education and model assessment.

Contribution

It presents a novel educational shared task on evaluating LLM outputs for clinical questions, including design considerations and student feedback.

Findings

Student feedback on task design

Insights into LLM output evaluation

Relevance for NLP education

Abstract

This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments.