LoraMap: Harnessing the Power of LoRA Connections
Hyeryun Park, Jeongwon Kwak, Dongsuk Jang, Sumin Park, Jinwook Choi

TL;DR
LoraMap is a novel method that establishes connections among multiple LoRA modules to improve fact-checking in Large Language Models, outperforming existing integration techniques with fewer parameters.
Contribution
This paper introduces LoraMap, a new approach to connect multiple LoRAs, enhancing fact-checking capabilities in LLMs beyond prior parallel integration methods.
Findings
LoraMap outperforms LoraHub in fact-checking tasks.
LoraMap requires fewer trainable parameters than LoraConcat.
LoraMap improves reasoning accuracy by connecting diverse LoRAs.
Abstract
Fact-checking techniques can mitigate hallucinations in Large Language Models (LLMs), a prominent issue in specialized domains. As parameter-efficient techniques such as Low-Rank Adaptation (LoRA) can overcome substantial computational overhead, some studies have explored the integration of multiple LoRAs. While previous studies focus on parallel integration, this paper investigates methods to establish connections among multiple LoRAs. We create three reasoning datasets tailored to fact-checking and fine-tune individual LoRAs, allowing them to view and reason from diverse perspectives. Then, we explore strategies for allocating these reasoning LoRAs and introduce LoraMap, an approach to map connections between them. The results of the fact-checking task demonstrate that the performance of LoraMap is superior to LoraHub, an existing method for integrating LoRAs. LoraMap also outperforms…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsICT in Developing Communities · IoT Networks and Protocols · Opportunistic and Delay-Tolerant Networks
MethodsFocus
