Massive Editing for Large Language Models Based on Dynamic Weight Generation
Wentao Wan, Qiqing Lao, Zhiwei Xie, Hefeng Wu, Runnan Lin, Liang Lin, Keze Wang
TL;DR
This paper introduces MeG, a novel method for large-scale knowledge editing in LLMs using dynamic weight generation conditioned on input queries, significantly improving edit reliability, generality, and locality.
Contribution
It proposes a dynamic weight neuron approach combined with a diffusion model to enable efficient large-scale knowledge editing in LLMs.
Findings
Significant improvement in Locality metric over existing methods
Enhanced Reliability and Generality in knowledge edits
Effective large-scale knowledge editing demonstrated
Abstract
Knowledge Editing (KE) is a field that studies how to modify some knowledge in Large Language Models (LLMs) at a low cost (compared to pre-training). Currently, performing large-scale edits on LLMs while ensuring the Reliability, Generality, and Locality metrics of the edits remain a challenge. This paper proposes a Massive editing approach for LLMs based on dynamic weight Generation (MeG). Our MeG involves attaching a dynamic weight neuron to specific layers of the LLMs and using a diffusion model to conditionally generate the weights of this neuron based on the input query required for the knowledge. This allows the use of adding a single dynamic weight neuron to achieve the goal of large-scale knowledge editing. Experiments show that our MeG can significantly improve the performance of large-scale KE in terms of Reliability, Generality, and Locality metrics compared to existing…
Peer Reviews
Decision·ICLR 2026 Poster
+ It is indeed interesting to see the paper reframes large-scale KE as conditional weight generation via diffusion for a single attachable neuron, which could address interference accumulation and capacity saturation in inner-weight editing and extra-neuron methods in a principled manner. + The authors have conducted extensive experiments with multiple backbone LLMs on multiple datasets, which demonstrate the effectiveness of the proposed method compared with the baselines.
- The method relies on a tuned text encoder, a K-way familiarity network with entropy thresholding, and a DiT-based generator, which is complex compared to the baselines. - The usage of the diffusion module seems not very well justified. I'm curious if we could directly learn a simpler mapping function between the original neuron and the new neuron instead of relying on the potentially unstable diffusion process.
- The paper departs from typical weight-modification-heavy or neuron-expansion approaches. The dynamic generation of a single neuron per query, via a diffusion model, is a creative application of generative modeling tailored for knowledge editing. This removes the static storage overhead of previous neural expansion methods and decouples editing capacity from LLM size. - The empirical validation, as shown in Table 1 and Table 2, covers different LLMs and datasets at large edit scales, with MeG c
- While the method is empirically validated, the theoretical underpinning regarding the maximum knowledge capacity, expressivity, and tradeoffs of allocating only one neuron per edit is weak. For example, in Section 4, while Equation (velocity prediction) and the overall diffusion objective are clearly described, no guarantees or fundamental analysis explain why or when this approach might fail, especially as the number or diversity of edits grows. Are there queries for which a single-neuron int
Please see above.
Please see above.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Advanced Graph Neural Networks · Natural Language Processing Techniques