Privately Learning from Graphs with Applications in Fine-tuning Large Language Models

TL;DR

This paper introduces a novel privacy-preserving method for relational learning on graphs, enabling the fine-tuning of large language models with sensitive graph data while balancing privacy, utility, and computational efficiency.

Contribution

It proposes a tailored DP-SGD pipeline that decouples dependencies in relational data, facilitating privacy-preserving fine-tuning of large language models on sensitive graphs.

Findings

Significant improvements in relational learning tasks with privacy guarantees

Effective fine-tuning of Llama2 on sensitive graph data

Analysis of privacy-utility-computation trade-offs

Abstract

Graphs offer unique insights into relationships between entities, complementing data modalities like text and images and enabling AI models to extend their capabilities beyond traditional tasks. However, learning from graphs often involves handling sensitive relationships in the data, raising significant privacy concerns. Existing privacy-preserving methods, such as DP-SGD, rely on gradient decoupling assumptions and are incompatible with relational learning due to the inherent dependencies between training samples. To address this challenge, we propose a privacy-preserving pipeline for relational learning that decouples dependencies in sampled relations for training, ensuring differential privacy through a tailored application of DP-SGD. We apply this approach to fine-tune large language models (LLMs), such as Llama2, on sensitive graph data while addressing the associated computational complexities. Our method is evaluated on four real-world text-attributed graphs, demonstrating significant improvements in relational learning tasks while maintaining robust privacy guarantees. Additionally, we analyze the trade-offs between privacy, utility, and computational efficiency, offering insights into the practical deployment of our approach for privacy-preserving relational learning. Code is available at https://github.com/Graph-COM/PvGaLM.