Distracting Downpour: Adversarial Weather Attacks for Motion Estimation

TL;DR

This paper introduces a novel adversarial attack on motion estimation that uses optimized particles to simulate realistic weather effects, revealing vulnerabilities and suggesting robustness improvements through weather augmentation.

Contribution

The work presents a differentiable particle rendering system for creating adversarial weather effects that impact motion estimation methods.

Findings

Adversarial weather significantly disrupts motion estimation.

Robustness improves with training augmented by non-optimized weather.

Methods previously resilient to pixel perturbations are vulnerable to weather attacks.

Abstract

Current adversarial attacks on motion estimation, or optical flow, optimize small per-pixel perturbations, which are unlikely to appear in the real world. In contrast, adverse weather conditions constitute a much more realistic threat scenario. Hence, in this work, we present a novel attack on motion estimation that exploits adversarially optimized particles to mimic weather effects like snowflakes, rain streaks or fog clouds. At the core of our attack framework is a differentiable particle rendering system that integrates particles (i) consistently over multiple time steps (ii) into the 3D space (iii) with a photo-realistic appearance. Through optimization, we obtain adversarial weather that significantly impacts the motion estimation. Surprisingly, methods that previously showed good robustness towards small per-pixel perturbations are particularly vulnerable to adversarial weather. At the same time, augmenting the training with non-optimized weather increases a method's robustness towards weather effects and improves generalizability at almost no additional cost. Our code will be available at https://github.com/cv-stuttgart/DistractingDownpour.