Optimising the Input Image to Improve Visual Relationship Detection

TL;DR

This paper investigates how different image preprocessing techniques affect visual relationship detection, finding that the Union-WB-and-B method significantly improves predicate prediction by enabling CNNs to better identify subjects and objects.

Contribution

The study introduces and evaluates alternative preprocessing methods, demonstrating that Union-WB-and-B enhances CNN performance in visual relationship detection.

Findings

Union-WB-and-B outperforms standard Union method

Preprocessing improves predicate prediction accuracy

CNNs can identify subjects and objects earlier in processing

Abstract

Visual Relationship Detection is defined as, given an image composed of a subject and an object, the correct relation is predicted. To improve the visual part of this difficult problem, ten preprocessing methods were tested to determine whether the widely used Union method yields the optimal results. Therefore, focusing solely on predicate prediction, no object detection and linguistic knowledge were used to prevent them from affecting the comparison results. Once fine-tuned, the Visual Geometry Group models were evaluated using Recall@1, per-predicate recall, activation maximisations, class activation maps, and error analysis. From this research it was found that using preprocessing methods such as the Union-Without-Background-and-with-Binary-mask (Union-WB-and-B) method yields significantly better results than the widely used Union method since, as designed, it enables the Convolutional Neural Network to also identify the subject and object in the convolutional layers instead of solely in the fully-connected layers.