IB-AdCSCNet:Adaptive Convolutional Sparse Coding Network Driven by Information Bottleneck

TL;DR

IB-AdCSCNet is a deep learning model that incorporates the information bottleneck principle with adaptive sparse coding, improving robustness and generalization by balancing compression and data fitting during training.

Contribution

The paper introduces IB-AdCSCNet, which adaptively adjusts the information bottleneck trade-off in deep networks using gradient descent within the FISTA framework, merging sparse coding with deep learning.

Findings

Matches residual networks on CIFAR-10/100

Outperforms residual networks on corrupted data

Enhances model robustness through IB trade-off inference

Abstract

In the realm of neural network models, the perpetual challenge remains in retaining task-relevant information while effectively discarding redundant data during propagation. In this paper, we introduce IB-AdCSCNet, a deep learning model grounded in information bottleneck theory. IB-AdCSCNet seamlessly integrates the information bottleneck trade-off strategy into deep networks by dynamically adjusting the trade-off hyperparameter $\lambda$ through gradient descent, updating it within the FISTA(Fast Iterative Shrinkage-Thresholding Algorithm ) framework. By optimizing the compressive excitation loss function induced by the information bottleneck principle, IB-AdCSCNet achieves an optimal balance between compression and fitting at a global level, approximating the globally optimal representation feature. This information bottleneck trade-off strategy driven by downstream tasks not only helps to learn effective features of the data, but also improves the generalization of the model. This study's contribution lies in presenting a model with consistent performance and offering a fresh perspective on merging deep learning with sparse representation theory, grounded in the information bottleneck concept. Experimental results on CIFAR-10 and CIFAR-100 datasets demonstrate that IB-AdCSCNet not only matches the performance of deep residual convolutional networks but also outperforms them when handling corrupted data. Through the inference of the IB trade-off, the model's robustness is notably enhanced.