TrustGLM: Evaluating the Robustness of GraphLLMs Against Prompt, Text, and Structure Attacks

TL;DR

This paper evaluates the robustness of GraphLLMs against adversarial attacks on text, structure, and prompts, revealing significant vulnerabilities and proposing defense strategies to improve their resilience.

Contribution

It introduces TrustGLM, a comprehensive framework for assessing and enhancing the robustness of GraphLLMs against various adversarial attacks.

Findings

GraphLLMs are highly vulnerable to text-based adversarial attacks.

Structural attack methods can significantly impair model performance.

Prompt shuffling causes notable performance degradation.

Abstract

Inspired by the success of large language models (LLMs), there is a significant research shift from traditional graph learning methods to LLM-based graph frameworks, formally known as GraphLLMs. GraphLLMs leverage the reasoning power of LLMs by integrating three key components: the textual attributes of input nodes, the structural information of node neighborhoods, and task-specific prompts that guide decision-making. Despite their promise, the robustness of GraphLLMs against adversarial perturbations remains largely unexplored-a critical concern for deploying these models in high-stakes scenarios. To bridge the gap, we introduce TrustGLM, a comprehensive study evaluating the vulnerability of GraphLLMs to adversarial attacks across three dimensions: text, graph structure, and prompt manipulations. We implement state-of-the-art attack algorithms from each perspective to rigorously assess model resilience. Through extensive experiments on six benchmark datasets from diverse domains, our findings reveal that GraphLLMs are highly susceptible to text attacks that merely replace a few semantically similar words in a node's textual attribute. We also find that standard graph structure attack methods can significantly degrade model performance, while random shuffling of the candidate label set in prompt templates leads to substantial performance drops. Beyond characterizing these vulnerabilities, we investigate defense techniques tailored to each attack vector through data-augmented training and adversarial training, which show promising potential to enhance the robustness of GraphLLMs. We hope that our open-sourced library will facilitate rapid, equitable evaluation and inspire further innovative research in this field.