Optimization of GNN Training Through Half-precision

TL;DR

HalfGNN introduces novel half-precision techniques for GNN training, significantly improving speed and memory efficiency while maintaining accuracy, addressing overflow and hardware under-utilization issues present in prior systems.

Contribution

The paper presents HalfGNN, a new GNN system utilizing half-precision with innovative vector operations and discretized SpMM to enhance performance and resource utilization.

Findings

2.30X average training speedup over DGL.

2.67X memory savings.

Maintains comparable accuracy to float-based methods.

Abstract

Recent trends in lower precision, e.g. half-precision floating point, training have shown improved system performance and reduced memory usage for Deep Learning while maintaining accuracy. However, current GNN systems cannot achieve such goals for GNN, as our analyses show that they massively underperform while showing abnormal accuracy when using half-precision. These systems suffer from value overflow issues due to lowered precision, under-utilization of hardware resources, and poor training performance. To mitigate this, we introduce HalfGNN, a half-precision based GNN system. HalfGNN proposes novel techniques: new vector operations for half-precision data types that improve data load and reduction performance, and discretized SpMM that overcomes the value overflow and natively provides workload balancing. Such techniques improve hardware utilization, reduce memory usage, and remove atomic writes. Evaluations show that HalfGNN achieves on average of 2.30X speedup in training time over DGL (float-based) for GAT, GCN, and GIN respectively while achieving similar accuracy, and saving 2.67X memory.