iShumei-Chinchunmei at SemEval-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning loss

TL;DR

This paper presents a multi-task framework with an effective unlearning loss to improve the efficiency and control of forgetting sensitive data in large language models, addressing a key challenge in machine unlearning.

Contribution

It introduces a novel effective unlearning loss and demonstrates its integration for balanced forgetting and retention in LLMs, advancing unlearning research.

Findings

Achieved competitive ranking in SemEval-2025 Task 4

Proposed a controllable unlearning loss for better data erasure

Enhanced efficiency in unlearning with minimal impact on model capabilities

Abstract

As the Large Language Model (LLM) gains widespread adoption, increasing attention has been given to the challenge of making LLM forget non-compliant data memorized during its pre-training. Machine Unlearning focuses on efficiently erasing sensitive information from LLM under limited computational resources. To advance research in this area, SemEval 2025 Task 4: "Unlearning Sensitive Content from Large Language Models" introduces three unlearning datasets and establishes a benchmark by evaluating both forgetting effectiveness and the preservation of standard capabilities. In this work, we propose a more controllable forgetting loss, Effective Unlearning Loss, and explore its integration with various techniques to achieve more efficient and controlled unlearning. Our system ultimately ranked 5th on the competition leaderboard.