# Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction

## TL;DR

This paper shows that modeling instance difficulty can enhance biomedical information extraction by improving training and routing, leveraging both universal and task-specific encoders, and effectively utilizing expert and lay annotations.

## Contribution

It introduces a difficulty prediction model combining universal and task-specific encoders to improve annotation strategies and model performance in biomedical NLP tasks.

## Key findings

- Excluding difficult instances slightly improves performance.
- Difficulty-based weighting during training yields consistent gains.
- Routing difficult instances to experts enhances annotation quality.

## Abstract

Modern NLP systems require high-quality annotated data. In specialized domains, expert annotations may be prohibitively expensive. An alternative is to rely on crowdsourcing to reduce costs at the risk of introducing noise. In this paper we demonstrate that directly modeling instance difficulty can be used to improve model performance, and to route instances to appropriate annotators. Our difficulty prediction model combines two learned representations: a `universal' encoder trained on out-of-domain data, and a task-specific encoder. Experiments on a complex biomedical information extraction task using expert and lay annotators show that: (i) simply excluding from the training data instances predicted to be difficult yields a small boost in performance; (ii) using difficulty scores to weight instances during training provides further, consistent gains; (iii) assigning instances predicted to be difficult to domain experts is an effective strategy for task routing. Our experiments confirm the expectation that for specialized tasks expert annotations are higher quality than crowd labels, and hence preferable to obtain if practical. Moreover, augmenting small amounts of expert data with a larger set of lay annotations leads to further improvements in model performance.