TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domain

TL;DR

TermGPT introduces a multi-level contrastive fine-tuning method that significantly improves domain-specific terminology representation in legal and financial language models, enhancing their discrimination ability for downstream tasks.

Contribution

The paper presents a novel multi-level contrastive fine-tuning framework and a new financial terminology dataset to improve terminology discrimination in LLMs for legal and financial domains.

Findings

Outperforms existing baselines in terminology discrimination tasks

Enhances semantic and structural understanding of domain-specific terms

Provides a new dataset for financial terminology evaluation

Abstract

Large language models (LLMs) have demonstrated impressive performance in text generation tasks; however, their embedding spaces often suffer from the isotropy problem, resulting in poor discrimination of domain-specific terminology, particularly in legal and financial contexts. This weakness in terminology-level representation can severely hinder downstream tasks such as legal judgment prediction or financial risk analysis, where subtle semantic distinctions are critical. To address this problem, we propose TermGPT, a multi-level contrastive fine-tuning framework designed for terminology adaptation. We first construct a sentence graph to capture semantic and structural relations, and generate semantically consistent yet discriminative positive and negative samples based on contextual and topological cues. We then devise a multi-level contrastive learning approach at both the sentence and token levels, enhancing global contextual understanding and fine-grained terminology discrimination. To support robust evaluation, we construct the first financial terminology dataset derived from official regulatory documents. Experiments show that TermGPT outperforms existing baselines in term discrimination tasks within the finance and legal domains.