PropMEND: Hypernetworks for Knowledge Propagation in LLMs

TL;DR

PropMEND introduces a hypernetwork-based method that enhances knowledge propagation in large language models, enabling better reasoning with injected facts, especially for multi-hop questions, and demonstrates improved performance on new datasets.

Contribution

The paper proposes PropMEND, a novel hypernetwork approach that extends meta-learning to improve knowledge propagation in LLMs, especially for multi-hop reasoning tasks.

Findings

Almost 2x accuracy on multi-hop questions in RippleEdit

Outperforms existing methods on unseen entity-relation pairs

Performance gap decreases on generalization, indicating room for improvement

Abstract

Knowledge editing techniques for large language models (LLMs) can inject knowledge that is later reproducible verbatim, but they fall short on propagating that knowledge: models cannot answer questions that require reasoning with the injected knowledge. We present a hypernetwork-based approach for knowledge propagation, named PropMEND, where we meta-learn how to modify gradients of a language modeling loss to encourage injected information to propagate. Our approach extends the meta-objective of MEND [29] so that gradient updates on knowledge are transformed to enable answering multi-hop questions involving that knowledge. We show improved performance on the RippleEdit dataset, showing almost 2x accuracy on challenging multi-hop questions whose answers are not explicitly stated in the injected fact. We further introduce a new dataset, Controlled RippleEdit, to evaluate the generalization of our hypernetwork, testing knowledge propagation along relations and entities unseen during hypernetwork training. PropMEND still outperforms existing approaches in unseen entity-relation pairs, yet the performance gap decreases substantially, suggesting future work in propagating knowledge to a wide range of relations.