Generation-Augmented Generation: A Plug-and-Play Framework for Private Knowledge Injection in Large Language Models

TL;DR

GAG is a novel framework that injects private domain knowledge into large language models by treating it as an auxiliary modality, improving specialist question-answering while maintaining general capabilities.

Contribution

It introduces Generation-Augmented Generation (GAG), a plug-and-play method that distills domain expertise into latent memories and integrates them into frozen models, outperforming existing baselines.

Findings

GAG outperforms retrieval-based and fine-tuning baselines on private-domain QA tasks.

GAG maintains general-domain performance while enhancing private knowledge injection.

Code and datasets are publicly available at https://github.com/360CVGroup/GAG.

Abstract

In domains such as materials science, biomedicine, and finance, high-stakes deployment of large language models (LLMs) requires injecting private, domain-specific knowledge that is proprietary, fast-evolving, and under-represented in public pretraining. However, the two dominant paradigms for private knowledge injection each have clear drawbacks: fine-tuning is expensive to iterate under continual updates that can induce catastrophic forgetting and general-capability regression; retrieval-augmented generation (RAG) keeps the base model intact but remains brittle in specialized private corpora due to chunk-induced evidence fragmentation, retrieval mismatch, and long-context pressure. Inspired by how multimodal LLMs align heterogeneous modalities into a shared semantic space, we propose Generation-Augmented Generation (GAG), which treats private expertise as an auxiliary modality and injects it into a frozen base model through a compact, constant-budget latent interface. Concretely, GAG distills question-conditioned specialist knowledge from lightweight domain experts into multi-slot latent memories, integrates multi-layer expert signals via per-slot cross-layer fusion, and aligns them to the frozen base model through gated residual projection, while supporting scalable mixed-domain deployment with reliable selective activation. In a unified mixed-domain evaluation spanning two scientific private-domain QA benchmarks (catalytic materials and immunology adjuvant) together with general-domain queries, GAG consistently outperforms strong retrieval-based and parameter-efficient fine-tuning baselines on specialist QA, while preserving general-domain capability, achieving highly reliable routing, and offering a favorable efficiency--effectiveness trade-off. Code and datasets are provided in the supplementary material. Code is publicly available at https://github.com/360CVGroup/GAG.