Enriched Annotations for Tumor Attribute Classification from Pathology Reports with Limited Labeled Data

TL;DR

This paper introduces a novel annotation scheme and algorithm that improve tumor attribute classification from pathology reports using limited labeled data, reducing annotation effort and achieving high accuracy with fewer labels.

Contribution

The authors develop an enriched hierarchical annotation scheme and the Supervised Line Attention algorithm, enabling effective tumor attribute classification with less labeled data and annotation time.

Findings

SLA achieves similar or better accuracy with half the labeled data compared to state-of-the-art methods.

Enriched annotations increase annotation time by 20%, but reduce overall annotation effort by 40%.

The method is effective on small datasets of 32 to 186 documents.

Abstract

Precision medicine has the potential to revolutionize healthcare, but much of the data for patients is locked away in unstructured free-text, limiting research and delivery of effective personalized treatments. Generating large annotated datasets for information extraction from clinical notes is often challenging and expensive due to the high level of expertise needed for high quality annotations. To enable natural language processing for small dataset sizes, we develop a novel enriched hierarchical annotation scheme and algorithm, Supervised Line Attention (SLA), and apply this algorithm to predicting categorical tumor attributes from kidney and colon cancer pathology reports from the University of California San Francisco (UCSF). Whereas previous work only annotated document level labels, we in addition ask the annotators to enrich the traditional label by asking them to also highlight the relevant line or potentially lines for the final label, which leads to a 20% increase of annotation time required per document. With the enriched annotations, we develop a simple and interpretable machine learning algorithm that first predicts the relevant lines in the document and then predicts the tumor attribute. Our results show across the small dataset sizes of 32, 64, 128, and 186 labeled documents per cancer, SLA only requires half the number of labeled documents as state-of-the-art methods to achieve similar or better micro-f1 and macro-f1 scores for the vast majority of comparisons that we made. Accounting for the increased annotation time, this leads to a 40% reduction in total annotation time over the state of the art.