"Why" Has the Least Side Effect on Model Editing

TL;DR

This paper investigates the factors influencing side effects in model editing of large language models, revealing that question type, model size, and batch size significantly affect performance and side effects.

Contribution

It categorizes model editing questions by type, compares effects across model sizes, and explores batch size impact, providing new insights into minimizing side effects.

Findings

Performance degradation varies by question type.

Insights from small models may not apply to large models.

Increasing batch size reduces side effects.

Abstract

Training large language models (LLMs) from scratch is an expensive endeavor, particularly as world knowledge continually evolves. To maintain relevance and accuracy of LLMs, model editing has emerged as a pivotal research area. While these methods hold promise, they can also produce unintended side effects. Their underlying factors and causes remain largely unexplored. This paper delves into a critical factor-question type-by categorizing model editing questions. Our findings reveal that the extent of performance degradation varies significantly across different question types, providing new insights for experimental design in knowledge editing. Furthermore, we investigate whether insights from smaller models can be extrapolated to larger models. Our results indicate discrepancies in findings between models of different sizes, suggesting that insights from smaller models may not necessarily apply to larger models. Additionally, we examine the impact of batch size on side effects, discovering that increasing the batch size can mitigate performance drops.