Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset

TL;DR

This study demonstrates that fine-tuning a large language model on a small Finnish medical transcription dataset can achieve meaningful semantic accuracy, supporting its use in low-resource clinical NLP tasks.

Contribution

It introduces a domain-specific fine-tuning approach for LLaMA 3.1-8B on Finnish medical transcription data, showing feasibility in low-resource settings.

Findings

Semantic similarity was high despite low n-gram overlap.

Fine-tuning improved model performance for Finnish clinical transcription.

Supports privacy-preserving NLP applications in healthcare.

Abstract

Clinical documentation is a critical factor for patient safety, diagnosis, and continuity of care. The administrative burden of EHRs is a significant factor in physician burnout. This is a critical issue for low-resource languages, including Finnish. This study aims to investigate the effectiveness of a domain-aligned natural language processing (NLP); large language model for medical transcription in Finnish by fine-tuning LLaMA 3.1-8B on a small validated corpus of simulated clinical conversations by students at Metropolia University of Applied Sciences. The fine-tuning process for medical transcription used a controlled preprocessing and optimization approach. The fine-tuning effectiveness was evaluated by sevenfold cross-validation. The evaluation metrics for fine-tuned LLaMA 3.1-8B were BLEU = 0.1214, ROUGE-L = 0.4982, and BERTScore F1 = 0.8230. The results showed a low n-gram overlap but a strong semantic similarity with reference transcripts. This study indicate that fine-tuning can be an effective approach for translation of medical discourse in spoken Finnish and support the feasibility of fine-tuning a privacy-oriented domain-specific large language model for clinical documentation in Finnish. Beside that provide directions for future work.