Per-parameter Task Arithmetic for Unlearning in Large Language Models

TL;DR

This paper introduces PerTA, a per-parameter adjustment method for task arithmetic in large language model unlearning, improving privacy information removal while preserving model utility.

Contribution

It proposes a novel per-parameter rescaling mechanism for task arithmetic, enhancing unlearning effectiveness and reducing over-forgetting in large language models.

Findings

PerTA outperforms standard task vectors in unlearning tasks.

PerTA surpasses training-based unlearning methods in effectiveness.

The method maintains model utility while improving privacy removal.

Abstract

In large language model (LLM) unlearning, private information is required to be removed. Task arithmetic unlearns by subtracting a specific task vector (TV)--defined as the parameter difference between a privacy-information-tuned model and the original model. While efficient, it can cause over-forgetting by disrupting parameters essential for retaining other information. Motivated by the observation that each parameter exhibits different importance for forgetting versus retention, we propose a per-parameter task arithmetic (PerTA) mechanism to rescale the TV, allowing per-parameter adjustment. These weights quantify the relative importance of each parameter for forgetting versus retention, estimated via gradients (i.e., PerTA-grad) or the diagonal Fisher information approximation (i.e., PerTA-fisher). Moreover, we discuss the effectiveness of PerTA, extend it to a more general form, and provide further analysis. Extensive experiments demonstrate that PerTA consistently improves upon standard TV, and in many cases surpasses widely used training-based unlearning methods in both forgetting effectiveness and overall model utility. By retaining the efficiency of task arithmetic while mitigating over-forgetting, PerTA offers a principled and practical framework for LLM unlearning.