Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs

TL;DR

This paper introduces LIBU, a lightweight influence-based unlearning algorithm enhanced with LoRA and second-order optimization, effectively removing specific knowledge from large language models without retraining.

Contribution

The paper presents LIBU, a novel unlearning method combining influence functions and second-order optimization, tailored for large language models, and demonstrates its effectiveness across various tasks.

Findings

Effective removal of specific knowledge from LLMs

Maintains overall utility of models after unlearning

Applicable to different types of tasks

Abstract

This paper describes LIBU (LoRA enhanced influence-based unlearning), an algorithm to solve the task of unlearning - removing specific knowledge from a large language model without retraining from scratch and compromising its overall utility (SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models). The algorithm combines classical \textit{influence functions} to remove the influence of the data from the model and \textit{second-order optimization} to stabilize the overall utility. Our experiments show that this lightweight approach is well applicable for unlearning LLMs in different kinds of task.