Selective Fine-Tuning of GPT Architectures for Parameter-Efficient Clinical Text Classification

TL;DR

This paper introduces a selective fine-tuning method for GPT-2 that efficiently adapts large language models to clinical text classification by updating only a small subset of parameters, achieving high accuracy with reduced computational costs.

Contribution

It proposes a parameter-efficient selective fine-tuning framework for GPT-2 tailored to clinical text classification, significantly reducing training resources while maintaining performance.

Findings

Achieves 91% classification accuracy on radiology reports.

Updates less than 6% of model parameters during training.

Outperforms head-only training and approaches full fine-tuning performance.

Abstract

The rapid expansion of electronic health record (EHR) systems has generated large volumes of unstructured clinical narratives that contain valuable information for disease identification, patient cohort discovery, and clinical decision support. Extracting structured knowledge from these free-text documents remains challenging because clinical language is highly specialized, labeled datasets are limited, and full fine-tuning of large pretrained language models can require substantial computational resources. Efficient adaptation strategies are therefore essential for practical clinical natural language processing applications. This study proposes a parameter-efficient selective fine-tuning framework for adapting GPT-2 to clinical text classification tasks. Instead of updating the entire pretrained model, the majority of network parameters are frozen, and only the final Transformer block, the final layer normalization module, and a lightweight classification head are updated during training. This design substantially reduces the number of trainable parameters while preserving the contextual representation capabilities learned during pretraining. The proposed approach is evaluated using radiology reports from the MIMIC-IV-Note dataset with automatically derived CheXpert-style labels. Experiments on 50,000 radiology reports demonstrate that selective fine-tuning achieves approximately 91% classification accuracy while updating fewer than 6% of the model parameters. Comparative experiments with head-only training and full-model fine-tuning show that the proposed method provides a favorable balance between predictive performance and computational efficiency. These results indicate that selective fine-tuning offers an efficient and scalable framework for clinical text classification.