Countering Inconsistent Labelling by Google's Vision API for Rotated Images

TL;DR

This paper demonstrates that Google's Vision API can be fooled by rotated images causing incorrect labels, but a pre-processing pipeline using a Res-Net50 model can correct orientation and improve label accuracy, enhancing robustness.

Contribution

Introduces a modular pre-processing pipeline with Res-Net50 to detect and correct image rotation, improving label prediction accuracy against adversarial rotations.

Findings

The pipeline accurately predicts image rotation angles.

Corrected images yield labels similar to correctly oriented images.

The approach significantly increases robustness to rotational adversarial examples.

Abstract

Google's Vision API analyses images and provides a variety of output predictions, one such type is context-based labelling. In this paper, it is shown that adversarial examples that cause incorrect label prediction and spoofing can be generated by rotating the images. Due to the black-boxed nature of the API, a modular context-based pre-processing pipeline is proposed consisting of a Res-Net50 model, that predicts the angle by which the image must be rotated to correct its orientation. The pipeline successfully performs the correction whilst maintaining the image's resolution and feeds it to the API which generates labels similar to the original correctly oriented image and using a Percentage Error metric, the performance of the corrected images as compared to its rotated counter-parts is found to be significantly higher. These observations imply that the API can benefit from such a pre-processing pipeline to increase robustness to rotational perturbances.