The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints

TL;DR

This study compares adapter techniques and lightweight Transformers for clinical text classification under strict resource constraints, finding that simpler models trained from scratch outperform adapter-augmented LLMs in low-data, low-compute settings.

Contribution

It provides a comprehensive evaluation of adapter methods versus lightweight Transformers for clinical NLP with limited data and hardware, highlighting the efficiency of training small models from scratch.

Findings

Lightweight Transformers outperform adapter-augmented LLMs under constraints.

Adapters, including GRN, do not consistently improve performance.

Small models trained from scratch require significantly less training time.

Abstract

Fine-tuning Large Language Models (LLMs) for clinical Natural Language Processing (NLP) poses significant challenges due to domain gap, limited data, and stringent hardware constraints. In this study, we evaluate four adapter techniques-Adapter, Lightweight, TinyAttention, and Gated Residual Network (GRN) - equivalent to Low-Rank Adaptation (LoRA), for clinical note classification under real-world, resource-constrained conditions. All experiments were conducted on a single NVIDIA Quadro P620 GPU (2 GB VRAM, 512 CUDA cores, 1.386 TFLOPS FP32), limiting batch sizes to <8 sequences and maximum sequence length to 256 tokens. Our clinical corpus comprises only 580 000 tokens, several orders of magnitude smaller than standard LLM pre-training datasets. We fine-tuned three biomedical pre-trained LLMs (CamemBERT-bio, AliBERT, DrBERT) and two lightweight Transformer models trained from scratch. Results show that 1) adapter structures provide no consistent gains when fine-tuning biomedical LLMs under these constraints, and 2) simpler Transformers, with minimal parameter counts and training times under six hours, outperform adapter-augmented LLMs, which required over 1000 GPU-hours. Among adapters, GRN achieved the best metrics (accuracy, precision, recall, F1 = 0.88). These findings demonstrate that, in low-resource clinical settings with limited data and compute, lightweight Transformers trained from scratch offer a more practical and efficient solution than large LLMs, while GRN remains a viable adapter choice when minimal adaptation is needed.