Sparsely Grouped Input Variables for Neural Networks

TL;DR

This paper introduces a novel method for inducing group sparsity in neural networks by using a new loss function and optimization algorithm, effectively selecting relevant input variable groups in real-world datasets.

Contribution

It proposes a new regularization technique and optimization approach for finding sparse input variable groups in multi-layer neural networks, extending beyond linear models.

Findings

Successfully achieved group sparsity in three real-world datasets.

Excluded significant portions of input variables while maintaining performance.

Demonstrated effectiveness in genomic, eye-tracking, and image recognition tasks.

Abstract

In genomic analysis, biomarker discovery, image recognition, and other systems involving machine learning, input variables can often be organized into different groups by their source or semantic category. Eliminating some groups of variables can expedite the process of data acquisition and avoid over-fitting. Researchers have used the group lasso to ensure group sparsity in linear models and have extended it to create compact neural networks in meta-learning. Different from previous studies, we use multi-layer non-linear neural networks to find sparse groups for input variables. We propose a new loss function to regularize parameters for grouped input variables, design a new optimization algorithm for this loss function, and test these methods in three real-world settings. We achieve group sparsity for three datasets, maintaining satisfying results while excluding one nucleotide position from an RNA splicing experiment, excluding 89.9% of stimuli from an eye-tracking experiment, and excluding 60% of image rows from an experiment on the MNIST dataset.