LAET: A Layer-wise Adaptive Ensemble Tuning Framework for Pretrained Language Models

TL;DR

LAET is a novel framework that selectively fine-tunes layers of pre-trained language models to reduce computational costs and improve performance in financial NLP tasks, outperforming larger models like GPT-4.

Contribution

Introduces Layer-wise Adaptive Ensemble Tuning (LAET), a method for efficient, layer-specific fine-tuning of LLMs tailored for financial NLP applications.

Findings

LAET reduces computational overhead significantly.

LAET outperforms existing benchmarks and state-of-the-art models.

Smaller LLMs with LAET achieve results comparable to larger models.

Abstract

Natural Language Processing (NLP) has transformed the financial industry, enabling advancements in areas such as textual analysis, risk management, and forecasting. Large language models (LLMs) like BloombergGPT and FinMA have set new benchmarks across various financial NLP tasks, including sentiment analysis, stock movement prediction, and credit risk assessment. Furthermore, FinMA-ES, a bilingual financial LLM, has also demonstrated strong performance using the FLARE and FLARE-ES benchmarks. However, the high computational demands of these models limit the accessibility of many organizations. To address this, we propose Layer-wise Adaptive Ensemble Tuning (LAET), a novel strategy that selectively fine-tunes the most effective layers of pre-trained LLMs by analyzing hidden state representations while freezing less critical layers. LAET significantly reduces computational overhead while enhancing task-specific performance. Our approach shows strong results in financial NLP tasks, outperforming existing benchmarks and state-of-the-art LLMs such as GPT-4, even with smaller LLMs ($\sim$3B parameters). This work bridges cutting-edge financial NLP research and real-world deployment with efficient and scalable models for financial applications.