Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM

TL;DR

This paper introduces a hypernetwork-driven meta-gated approach to enhance LLM adaptability across various textual conditions, outperforming finetuning and meta-learning methods.

Contribution

It proposes a novel meta-gating mechanism activated within SwiGLU blocks, enabling dynamic control of LLMs via a hypernetwork that adjusts based on textual conditions.

Findings

Outperforms finetuning and meta-learning baselines.

Generalizes well to unseen tasks and conditions.

Provides meta-controllability on LLMs.

Abstract

Conventional LLMs may suffer from corpus heterogeneity and subtle condition changes. While finetuning can create the catastrophe forgetting issue, application of meta-learning on LLMs is also limited due to its complexity and scalability. In this paper, we activate the meta-signal of $\beta$ within the SwiGLU blocks, resulting in a meta-gating mechanism that adaptively adjusts the nonlinearity of FFN. A hypernetwork is employed which dynamically produces $\beta$ on textual conditions, providing meta-controllability on LLMs. By testing on different condition types such as task, domain, persona, and style, our method outperforms finetuning and meta-learning baselines, and can generalize reasonably on unseen tasks, condition types, or instructions. Our code can be found in https://github.com/AaronJi/MeGan.