Guide Me: Interacting with Deep Networks

TL;DR

This paper introduces a method for guiding trained convolutional neural networks using natural language input, enhancing their performance during inference by inserting a spatio-semantic guide layer that adapts activations based on user queries.

Contribution

It proposes a novel approach to interactively guide CNNs with language, using automatic learning of verbal interactions without manual annotations, and demonstrates performance improvements.

Findings

Guided CNNs show improved accuracy on two datasets.

The method allows flexible, language-based control of neural network inference.

Automatic learning of verbal interaction does not require manual text annotations.

Abstract

Interaction and collaboration between humans and intelligent machines has become increasingly important as machine learning methods move into real-world applications that involve end users. While much prior work lies at the intersection of natural language and vision, such as image captioning or image generation from text descriptions, less focus has been placed on the use of language to guide or improve the performance of a learned visual processing algorithm. In this paper, we explore methods to flexibly guide a trained convolutional neural network through user input to improve its performance during inference. We do so by inserting a layer that acts as a spatio-semantic guide into the network. This guide is trained to modify the network's activations, either directly via an energy minimization scheme or indirectly through a recurrent model that translates human language queries to interaction weights. Learning the verbal interaction is fully automatic and does not require manual text annotations. We evaluate the method on two datasets, showing that guiding a pre-trained network can improve performance, and provide extensive insights into the interaction between the guide and the CNN.