LoraMap: Harnessing the Power of LoRA Connections

TL;DR

LoraMap is a novel method that establishes connections among multiple LoRA modules to improve fact-checking in Large Language Models, outperforming existing integration techniques with fewer parameters.

Contribution

This paper introduces LoraMap, a new approach to connect multiple LoRAs, enhancing fact-checking capabilities in LLMs beyond prior parallel integration methods.

Findings

LoraMap outperforms LoraHub in fact-checking tasks.

LoraMap requires fewer trainable parameters than LoraConcat.

LoraMap improves reasoning accuracy by connecting diverse LoRAs.

Abstract

Fact-checking techniques can mitigate hallucinations in Large Language Models (LLMs), a prominent issue in specialized domains. As parameter-efficient techniques such as Low-Rank Adaptation (LoRA) can overcome substantial computational overhead, some studies have explored the integration of multiple LoRAs. While previous studies focus on parallel integration, this paper investigates methods to establish connections among multiple LoRAs. We create three reasoning datasets tailored to fact-checking and fine-tune individual LoRAs, allowing them to view and reason from diverse perspectives. Then, we explore strategies for allocating these reasoning LoRAs and introduce LoraMap, an approach to map connections between them. The results of the fact-checking task demonstrate that the performance of LoraMap is superior to LoraHub, an existing method for integrating LoRAs. LoraMap also outperforms with significantly fewer trainable parameters than LoraConcat, which concatenates LoRAs and further fine-tunes them.