Why do linear SVMs trained on HOG features perform so well?

TL;DR

This paper investigates why linear SVMs trained on HOG features excel in visual tasks, revealing that HOG features enhance capacity and incorporate priors, especially through local second-order interactions, leading to high accuracy in recognition tasks.

Contribution

The paper provides a new perspective connecting feature extraction and learning, highlighting the importance of local second-order interactions in HOG features for SVM performance.

Findings

HOG features induce capacity and add priors to linear SVMs.

Preserving second-order statistics and locality improves performance.

Surprising accuracy achieved on expression recognition and pedestrian detection.

Abstract

Linear Support Vector Machines trained on HOG features are now a de facto standard across many visual perception tasks. Their popularisation can largely be attributed to the step-change in performance they brought to pedestrian detection, and their subsequent successes in deformable parts models. This paper explores the interactions that make the HOG-SVM symbiosis perform so well. By connecting the feature extraction and learning processes rather than treating them as disparate plugins, we show that HOG features can be viewed as doing two things: (i) inducing capacity in, and (ii) adding prior to a linear SVM trained on pixels. From this perspective, preserving second-order statistics and locality of interactions are key to good performance. We demonstrate surprising accuracy on expression recognition and pedestrian detection tasks, by assuming only the importance of preserving such local second-order interactions.