Divided We Fall: Defending Against Adversarial Attacks via Soft-Gated Fractional Mixture-of-Experts with Randomized Adversarial Training

TL;DR

This paper introduces a robust defense mechanism against adversarial attacks in machine learning, utilizing a mixture-of-experts architecture with adversarial training to improve resilience against white-box attacks on image classification tasks.

Contribution

It proposes a novel mixture-of-experts defense system with adversarial training, jointly optimizing experts and gating to enhance robustness against white-box attacks.

Findings

Outperforms prior MoE defenses under FGSM and PGD attacks

Uses nine pre-trained ResNet-18 experts for improved robustness

Inference cost scales linearly with the number of experts

Abstract

Machine learning is a powerful tool enabling full automation of a huge number of tasks without explicit programming. Despite recent progress of machine learning in different domains, these models have shown vulnerabilities when they are exposed to adversarial threats. Adversarial threats aim to hinder the machine learning models from satisfying their objectives. They can create adversarial perturbations, which are imperceptible to humans' eyes but have the ability to cause misclassification during inference. In this paper, we propose a defense system, which devises an adversarial training module within mixture-of-experts architecture to enhance its robustness against white-box evasion attacks. In our proposed defense system, we use nine pre-trained classifiers (experts) with ResNet-18 as their backbone. During end-to-end training, the parameters of all experts and the gating mechanism are jointly updated allowing further optimization of the experts. Our proposed defense system outperforms prior MoE-based defenses under strong white-box FGSM and PGD evaluation on CIFAR-10 and SVHN. The use of multiple experts increases training time and compute relative to single-network baselines; however, inference scales approximately linearly with the number of experts and is substantially cheaper than training.