SVM-based Deep Stacking Networks

TL;DR

This paper introduces a novel deep learning architecture called SVM-DSN, which combines the Deep Stacking Network framework with SVM classifiers, offering improved interpretability and performance on image and text data.

Contribution

The paper proposes integrating SVMs into the DSN architecture with a layer tuning scheme, enhancing interpretability and optimization in deep stacking networks.

Findings

SVM-DSN outperforms benchmark models on image and text datasets.

The model demonstrates improved anti-saturation properties.

Supports parallel training of deep SVM layers.

Abstract

The deep network model, with the majority built on neural networks, has been proved to be a powerful framework to represent complex data for high performance machine learning. In recent years, more and more studies turn to nonneural network approaches to build diverse deep structures, and the Deep Stacking Network (DSN) model is one of such approaches that uses stacked easy-to-learn blocks to build a parameter-training-parallelizable deep network. In this paper, we propose a novel SVM-based Deep Stacking Network (SVM-DSN), which uses the DSN architecture to organize linear SVM classifiers for deep learning. A BP-like layer tuning scheme is also proposed to ensure holistic and local optimizations of stacked SVMs simultaneously. Some good math properties of SVM, such as the convex optimization, is introduced into the DSN framework by our model. From a global view, SVM-DSN can iteratively extract data representations layer by layer as a deep neural network but with parallelizability, and from a local view, each stacked SVM can converge to its optimal solution and obtain the support vectors, which compared with neural networks could lead to interesting improvements in anti-saturation and interpretability. Experimental results on both image and text data sets demonstrate the excellent performances of SVM-DSN compared with some competitive benchmark models.