SimCalib: Graph Neural Network Calibration based on Similarity between Nodes

TL;DR

This paper introduces SimCalib, a novel GNN calibration framework that leverages node similarity at global and local levels, supported by theoretical analysis and extensive experiments showing state-of-the-art results.

Contribution

The work provides the first theoretical analysis linking GNN calibration to node similarity and proposes SimCalib, a new calibration method considering global and local similarity measures.

Findings

SimCalib achieves state-of-the-art calibration on 14 out of 16 benchmarks.

A correlation exists between node similarity and calibration improvement.

Theoretical analysis connects over-smoothing with calibration issues.

Abstract

Graph neural networks (GNNs) have exhibited impressive performance in modeling graph data as exemplified in various applications. Recently, the GNN calibration problem has attracted increasing attention, especially in cost-sensitive scenarios. Previous work has gained empirical insights on the issue, and devised effective approaches for it, but theoretical supports still fall short. In this work, we shed light on the relationship between GNN calibration and nodewise similarity via theoretical analysis. A novel calibration framework, named SimCalib, is accordingly proposed to consider similarity between nodes at global and local levels. At the global level, the Mahalanobis distance between the current node and class prototypes is integrated to implicitly consider similarity between the current node and all nodes in the same class. At the local level, the similarity of node representation movement dynamics, quantified by nodewise homophily and relative degree, is considered. Informed about the application of nodewise movement patterns in analyzing nodewise behavior on the over-smoothing problem, we empirically present a possible relationship between over-smoothing and GNN calibration problem. Experimentally, we discover a correlation between nodewise similarity and model calibration improvement, in alignment with our theoretical results. Additionally, we conduct extensive experiments investigating different design factors and demonstrate the effectiveness of our proposed SimCalib framework for GNN calibration by achieving state-of-the-art performance on 14 out of 16 benchmarks.