Deep Double Sparsity Encoder: Learning to Sparsify Not Only Features But Also Parameters

TL;DR

This paper introduces the Deep Double Sparsity Encoder (DDSE), a novel deep learning model that sparsifies both features and parameters, leading to compactness and superior performance, with promising applications in brain encoding.

Contribution

The paper proposes DDSE, a unified framework that jointly sparsifies features and parameters, inspired by double sparsity models, enhancing model efficiency and interpretability.

Findings

DDSE outperforms baseline models in various tasks.

DDSE achieves a compact model size with low complexity.

Preliminary results show promise in brain encoding applications.

Abstract

This paper emphasizes the significance to jointly exploit the problem structure and the parameter structure, in the context of deep modeling. As a specific and interesting example, we describe the deep double sparsity encoder (DDSE), which is inspired by the double sparsity model for dictionary learning. DDSE simultaneously sparsities the output features and the learned model parameters, under one unified framework. In addition to its intuitive model interpretation, DDSE also possesses compact model size and low complexity. Extensive simulations compare DDSE with several carefully-designed baselines, and verify the consistently superior performance of DDSE. We further apply DDSE to the novel application domain of brain encoding, with promising preliminary results achieved.