SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass

TL;DR

SHINE introduces a scalable hypernetwork that efficiently maps diverse contexts into high-quality LoRA adapters for large language models, enabling in-context knowledge transfer without fine-tuning.

Contribution

It presents a novel in-context hypernetwork architecture that generates LoRA adapters in a single pass, overcoming prior limitations and reducing resource costs.

Findings

Achieves strong performance on various tasks.

Reduces time, computation, and memory costs.

Enables complex question answering without direct context access.

Abstract

We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/MuLabPKU/SHINE