Diagnosing Medical Datasets with Training Dynamics

TL;DR

This paper investigates the use of training dynamics via Data Maps to evaluate medical datasets, finding that the framework is not suitable for the unique challenges of medical question answering.

Contribution

It assesses the transferability of Data Maps for dataset diagnosis in the medical domain, revealing limitations in medical question answering tasks.

Findings

Data Maps framework is unsuitable for medical datasets

Medical question answering requires specialized data evaluation methods

Training dynamics may not generalize across domains

Abstract

This study explores the potential of using training dynamics as an automated alternative to human annotation for evaluating the quality of training data. The framework used is Data Maps, which classifies data points into categories such as easy-to-learn, hard-to-learn, and ambiguous (Swayamdipta et al., 2020). Swayamdipta et al. (2020) highlight that difficult-to-learn examples often contain errors, and ambiguous cases significantly impact model training. To confirm the reliability of these findings, we replicated the experiments using a challenging dataset, with a focus on medical question answering. In addition to text comprehension, this field requires the acquisition of detailed medical knowledge, which further complicates the task. A comprehensive evaluation was conducted to assess the feasibility and transferability of the Data Maps framework to the medical domain. The evaluation indicates that the framework is unsuitable for addressing datasets' unique challenges in answering medical questions.