Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations

TL;DR

This paper explores domain-specific fine-tuning of Finnish BERT on medical texts and investigates how embedding changes can predict benefits in healthcare NLP tasks with limited labeled data.

Contribution

It provides insights into fine-tuning Finnish BERT on medical data and proposes methods to predict pre-training benefits from embedding geometry analysis.

Findings

Embedding changes correlate with downstream task improvements.

Domain fine-tuning enhances Finnish BERT's performance on medical text classification.

Embedding geometry can serve as a predictor for pre-training benefits.

Abstract

In NLP classification tasks where little labeled data exists, domain fine-tuning of transformer models on unlabeled data is an established approach. In this paper we have two aims. (1) We describe our observations from fine-tuning the Finnish BERT model on Finnish medical text data. (2) We report on our attempts to predict the benefit of domain-specific pre-training of Finnish BERT from observing the geometry of embedding changes due to domain fine-tuning. Our driving motivation is the common\situation in healthcare AI where we might experience long delays in acquiring datasets, especially with respect to labels.