Deep Fishing: Gradient Features from Deep Nets

TL;DR

This paper introduces a novel method that derives gradient-based features from convolutional networks, bridging deep learning and traditional Fisher Vector encoding, leading to improved image recognition performance.

Contribution

It presents a new way to extract gradient features from ConvNets, connecting deep learning with Fisher Vector encoding for enhanced image recognition.

Findings

Gradient features from ConvNets improve recognition accuracy.

Structured matrix representation enables efficient similarity computation.

Consistent performance gains on Pascal VOC datasets.

Abstract

Convolutional Networks (ConvNets) have recently improved image recognition performance thanks to end-to-end learning of deep feed-forward models from raw pixels. Deep learning is a marked departure from the previous state of the art, the Fisher Vector (FV), which relied on gradient-based encoding of local hand-crafted features. In this paper, we discuss a novel connection between these two approaches. First, we show that one can derive gradient representations from ConvNets in a similar fashion to the FV. Second, we show that this gradient representation actually corresponds to a structured matrix that allows for efficient similarity computation. We experimentally study the benefits of transferring this representation over the outputs of ConvNet layers, and find consistent improvements on the Pascal VOC 2007 and 2012 datasets.