Zero Memory Overhead Approach for Protecting Vision Transformer Parameters

TL;DR

This paper presents a zero memory overhead fault tolerance method for Vision Transformers that detects and masks bit-flip faults by using parity bits in least significant bits, significantly improving robustness in safety-critical applications.

Contribution

It introduces a novel zero-overhead fault detection and masking technique for ViT parameters using parity bits, enhancing reliability without increasing memory usage.

Findings

Improves fault tolerance of ViT models by up to three orders of magnitude.

Detects bit-flip faults with zero memory overhead using parity bits.

Effectively prevents accuracy degradation in safety-critical applications.

Abstract

Vision Transformers (ViTs) have demonstrated superior performance over Convolutional Neural Networks (CNNs) in various vision-related tasks such as classification, object detection, and segmentation due to their use of self-attention mechanisms. As ViTs become more popular in safety-critical applications like autonomous driving, ensuring their correct functionality becomes essential, especially in the presence of bit-flip faults in their parameters stored in memory. In this paper, a fault tolerance technique is introduced to protect ViT parameters against bit-flip faults with zero memory overhead. Since the least significant bits of parameters are not critical for model accuracy, replacing the LSB with a parity bit provides an error detection mechanism without imposing any overhead on the model. When faults are detected, affected parameters are masked by zeroing out, as most parameters in ViT models are near zero, effectively preventing accuracy degradation. This approach enhances reliability across ViT models, improving the robustness of parameters to bit-flips by up to three orders of magnitude, making it an effective zero-overhead solution for fault tolerance in critical applications.