Comparative Analysis of LoRA-Adapted Embedding Models for Clinical Cardiology Text Representation

TL;DR

This study systematically compares ten transformer-based models adapted for cardiology text using LoRA fine-tuning, revealing that smaller encoder-only models like BioLinkBERT outperform larger models in domain-specific tasks with less resource use.

Contribution

It provides the first comprehensive comparison of LoRA-adapted transformer models for clinical cardiology text representation, challenging assumptions about model size and performance.

Findings

Encoder-only models like BioLinkBERT outperform larger models.

Smaller models require less computational resources.

Results are publicly available for reproducibility.

Abstract

Domain-specific text embeddings are critical for clinical natural language processing, yet systematic comparisons across model architectures remain limited. This study evaluates ten transformer-based embedding models adapted for cardiology through Low-Rank Adaptation (LoRA) fine-tuning on 106,535 cardiology text pairs derived from authoritative medical textbooks. Results demonstrate that encoder-only architectures, particularly BioLinkBERT, achieve superior domain-specific performance (separation score: 0.510) compared to larger decoder-based models, while requiring significantly fewer computational resources. The findings challenge the assumption that larger language models necessarily produce better domain-specific embeddings and provide practical guidance for clinical NLP system development. All models, training code, and evaluation datasets are publicly available to support reproducible research in medical informatics.