Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language Models

TL;DR

Oblivionis is a lightweight framework that enables federated large language models to selectively forget private data, improving privacy compliance without sacrificing model performance.

Contribution

It introduces a dual optimization approach unifying federated learning and unlearning, with comprehensive algorithms for effective data removal in federated LLMs.

Findings

Outperforms local training in forgetting and utility balance

Provides a robust pipeline for federated LLM unlearning

Offers comparative analysis of multiple algorithms

Abstract

Large Language Models (LLMs) increasingly leverage Federated Learning (FL) to utilize private, task-specific datasets for fine-tuning while preserving data privacy. However, while federated LLM frameworks effectively enable collaborative training without raw data sharing, they critically lack built-in mechanisms for regulatory compliance like GDPR's right to be forgotten. Integrating private data heightens concerns over data quality and long-term governance, yet existing distributed training frameworks offer no principled way to selectively remove specific client contributions post-training. Due to distributed data silos, stringent privacy constraints, and the intricacies of interdependent model aggregation, federated LLM unlearning is significantly more complex than centralized LLM unlearning. To address this gap, we introduce Oblivionis, a lightweight learning and unlearning framework that enables clients to selectively remove specific private data during federated LLM training, enhancing trustworthiness and regulatory compliance. By unifying FL and unlearning as a dual optimization objective, we incorporate 6 FL and 5 unlearning algorithms for comprehensive evaluation and comparative analysis, establishing a robust pipeline for federated LLM unlearning. Extensive experiments demonstrate that Oblivionis outperforms local training, achieving a robust balance between forgetting efficacy and model utility, with cross-algorithm comparisons providing clear directions for future LLM development.