Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches

TL;DR

This paper compares embedding-based and instruction-based fine-tuning methods for large language models in text classification, demonstrating that embedding-based approaches outperform instruction-tuning and are resource-efficient.

Contribution

It introduces a resource-efficient fine-tuning strategy combining 4-bit quantization and LoRA, and shows embedding-based methods outperform instruction-tuning in classification tasks.

Findings

Embedding-based fine-tuning outperforms instruction-based methods in F1-score.

The combined approach achieves competitive results with domain-specific models.

Efficient fine-tuning enables high performance on large models with limited resources.

Abstract

We explore efficient strategies to fine-tune decoder-only Large Language Models (LLMs) for downstream text classification under resource constraints. Two approaches are investigated: (1) attaching a classification head to a pre-trained causal LLM and fine-tuning on the task (using the LLM's final token embedding as a sequence representation), and (2) instruction-tuning the LLM in a prompt->response format for classification. To enable single-GPU fine-tuning of models up to 8B parameters, we combine 4-bit model quantization with Low-Rank Adaptation (LoRA) for parameter-efficient training. Experiments on two datasets - a proprietary single-label dataset and the public WIPO-Alpha patent dataset (extreme multi-label classification) - show that the embedding-based method significantly outperforms the instruction-tuned method in F1-score, and is very competitive with - even surpassing - fine-tuned domain-specific models (e.g. BERT) on the same tasks. These results demonstrate that directly leveraging the internal representations of causal LLMs, along with efficient fine-tuning techniques, yields impressive classification performance under limited computational resources. We discuss the advantages of each approach while outlining practical guidelines and future directions for optimizing LLM fine-tuning in classification scenarios.