# Learning Sparse Neural Networks via $\ell_0$ and T$\ell_1$ by a Relaxed Variable Splitting Method with Application to Multi-scale Curve Classification

## TL;DR

This paper introduces a relaxed variable splitting method to sparsify convolutional neural networks using $0$ and Tb1 penalties, achieving high accuracy with significant weight reduction, especially in complex curve classification tasks.

## Contribution

The paper presents a novel optimization approach for neural network sparsification using $0$ and Tb1 penalties, demonstrating effective pruning in CNNs for complex curve classification.

## Key findings

- Achieved over 99% test accuracy with 86% sparsity in fully connected layer.
- Comparable sparsity and accuracy with both $0$ and Tb1 penalties.
- Effective classification of shaky vs. regular fonts and handwriting.

## Abstract

We study sparsification of convolutional neural networks (CNN) by a relaxed variable splitting method of $\ell_0$ and transformed-$\ell_1$ (T$\ell_1$) penalties, with application to complex curves such as texts written in different fonts, and words written with trembling hands simulating those of Parkinson's disease patients. The CNN contains 3 convolutional layers, each followed by a maximum pooling, and finally a fully connected layer which contains the largest number of network weights. With $\ell_0$ penalty, we achieved over 99 \% test accuracy in distinguishing shaky vs. regular fonts or hand writings with above 86 \% of the weights in the fully connected layer being zero. Comparable sparsity and test accuracy are also reached with a proper choice of T$\ell_1$ penalty.