L3iTC at the FinLLM Challenge Task: Quantization for Financial Text Classification & Summarization

TL;DR

This paper describes the L3iTC team's participation in the FinLLM Challenge 2024, where they used quantization and LoRA techniques to efficiently fine-tune large language models for financial text classification and summarization, achieving top rankings.

Contribution

The paper introduces the application of 4-bit quantization and LoRA for efficient LLM fine-tuning in financial NLP tasks, demonstrating competitive performance with reduced resource requirements.

Findings

Achieved third place in financial text classification with an F1-score of 0.7543.

Secured sixth place in financial text summarization.

Demonstrated effective use of quantization and LoRA for resource-efficient model training.

Abstract

This article details our participation (L3iTC) in the FinLLM Challenge Task 2024, focusing on two key areas: Task 1, financial text classification, and Task 2, financial text summarization. To address these challenges, we fine-tuned several large language models (LLMs) to optimize performance for each task. Specifically, we used 4-bit quantization and LoRA to determine which layers of the LLMs should be trained at a lower precision. This approach not only accelerated the fine-tuning process on the training data provided by the organizers but also enabled us to run the models on low GPU memory. Our fine-tuned models achieved third place for the financial classification task with an F1-score of 0.7543 and secured sixth place in the financial summarization task on the official test datasets.