SemAug: Semantically Meaningful Image Augmentations for Object Detection Through Language Grounding

TL;DR

SemAug introduces a novel image augmentation technique that injects semantically meaningful objects into scenes using language grounding, enhancing object detection performance without significant overhead.

Contribution

It proposes a new augmentation method that adds contextually relevant objects into images, improving generalization for object detection without extra training overhead.

Findings

Achieved 2-4% mAP improvement on Pascal VOC.

Achieved 1-2% mAP improvement on COCO.

Effective across various model architectures.

Abstract

Data augmentation is an essential technique in improving the generalization of deep neural networks. The majority of existing image-domain augmentations either rely on geometric and structural transformations, or apply different kinds of photometric distortions. In this paper, we propose an effective technique for image augmentation by injecting contextually meaningful knowledge into the scenes. Our method of semantically meaningful image augmentation for object detection via language grounding, SemAug, starts by calculating semantically appropriate new objects that can be placed into relevant locations in the image (the what and where problems). Then it embeds these objects into their relevant target locations, thereby promoting diversity of object instance distribution. Our method allows for introducing new object instances and categories that may not even exist in the training set. Furthermore, it does not require the additional overhead of training a context network, so it can be easily added to existing architectures. Our comprehensive set of evaluations showed that the proposed method is very effective in improving the generalization, while the overhead is negligible. In particular, for a wide range of model architectures, our method achieved ~2-4% and ~1-2% mAP improvements for the task of object detection on the Pascal VOC and COCO datasets, respectively.