SHIFT3D: Synthesizing Hard Inputs For Tricking 3D Detectors

TL;DR

SHIFT3D is a differentiable pipeline that generates challenging 3D shapes to identify vulnerabilities in 3D detectors, enhancing safety in autonomous systems by revealing potential failure modes.

Contribution

The paper introduces a novel method using SDF representations and gradient-based deformation to synthesize plausible yet difficult 3D objects for detector testing.

Findings

Generates physically different but semantically recognizable challenging objects.

Provides interpretable failure modes for 3D detectors.

Aids in preemptive safety risk discovery.

Abstract

We present SHIFT3D, a differentiable pipeline for generating 3D shapes that are structurally plausible yet challenging to 3D object detectors. In safety-critical applications like autonomous driving, discovering such novel challenging objects can offer insight into unknown vulnerabilities of 3D detectors. By representing objects with a signed distanced function (SDF), we show that gradient error signals allow us to smoothly deform the shape or pose of a 3D object in order to confuse a downstream 3D detector. Importantly, the objects generated by SHIFT3D physically differ from the baseline object yet retain a semantically recognizable shape. Our approach provides interpretable failure modes for modern 3D object detectors, and can aid in preemptive discovery of potential safety risks within 3D perception systems before these risks become critical failures.