Understanding LoRA as Knowledge Memory: An Empirical Analysis

TL;DR

This paper systematically investigates LoRA as a modular knowledge memory for LLMs, analyzing its capacity, scalability, and reasoning abilities to complement existing inference-time methods.

Contribution

It provides the first comprehensive empirical study of LoRA as a knowledge memory, exploring its design space and operational boundaries.

Findings

LoRA can effectively serve as a modular knowledge memory for LLMs.

Scaling multi-module LoRA systems enhances long-context reasoning.

LoRA offers advantages complementary to RAG and ICL methods.

Abstract

Continuous knowledge updating for pre-trained large language models (LLMs) is increasingly necessary yet remains challenging. Although inference-time methods like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) are popular, they face constraints in context budgets, costs, and retrieval fragmentation. Departing from these context-dependent paradigms, this work investigates a parametric approach using Low-Rank Adaptation (LoRA) as a modular knowledge memory. Although few recent works examine this concept, the fundamental mechanics governing its capacity and composability remain largely unexplored. We bridge this gap through the first systematic empirical study mapping the design space of LoRA-based memory, ranging from characterizing storage capacity and optimizing internalization to scaling multi-module systems and evaluating long-context reasoning. Rather than proposing a single architecture, we provide practical guidance on the operational boundaries of LoRA memory. Overall, our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.