Probing the Intra-Component Correlations within Fisher Vector for Material Classification

TL;DR

This paper introduces the Completed Fisher vector (CFV), a generalization of the Fisher vector that encodes intra-component correlations of local descriptors, leading to improved material classification performance.

Contribution

The paper proposes the CFV, which relaxes the decorrelation assumption of FV by including correlations, enhancing discriminative power in image representation.

Findings

CFV outperforms FV across all tested parameters

CFV is robust to the number of mixture components

CFV performs well with small visual vocabularies on challenging datasets

Abstract

Fisher vector (FV) has become a popular image representation. One notable underlying assumption of the FV framework is that local descriptors are well decorrelated within each cluster so that the covariance matrix for each Gaussian can be simplified to be diagonal. Though the FV usually relies on the Principal Component Analysis (PCA) to decorrelate local features, the PCA is applied to the entire training data and hence it only diagonalizes the \textit{universal} covariance matrix, rather than those w.r.t. the local components. As a result, the local decorrelation assumption is usually not supported in practice. To relax this assumption, this paper proposes a completed model of the Fisher vector, which is termed as the Completed Fisher vector (CFV). The CFV is a more general framework of the FV, since it encodes not only the variances but also the correlations of the whitened local descriptors. The CFV thus leads to improved discriminative power. We take the task of material categorization as an example and experimentally show that: 1) the CFV outperforms the FV under all parameter settings; 2) the CFV is robust to the changes in the number of components in the mixture; 3) even with a relatively small visual vocabulary the CFV still works well on two challenging datasets.