Convolution in Convolution for Network in Network

TL;DR

This paper introduces CiC, a sparse shallow MLP approach replacing dense MLPs in Network in Network architectures, improving feature representation while reducing parameters, validated on CIFAR datasets.

Contribution

Proposes a sparse shallow MLP (CiC) to replace dense MLPs in NiN, enhancing efficiency and effectiveness of feature learning.

Findings

CiC outperforms traditional NiN on CIFAR datasets.

Reduced parameters with maintained or improved accuracy.

Effective in both standard and augmented datasets.

Abstract

Network in Netwrok (NiN) is an effective instance and an important extension of Convolutional Neural Network (CNN) consisting of alternating convolutional layers and pooling layers. Instead of using a linear filter for convolution, NiN utilizes shallow MultiLayer Perceptron (MLP), a nonlinear function, to replace the linear filter. Because of the powerfulness of MLP and $ 1\times 1 $ convolutions in spatial domain, NiN has stronger ability of feature representation and hence results in better recognition rate. However, MLP itself consists of fully connected layers which give rise to a large number of parameters. In this paper, we propose to replace dense shallow MLP with sparse shallow MLP. One or more layers of the sparse shallow MLP are sparely connected in the channel dimension or channel-spatial domain. The proposed method is implemented by applying unshared convolution across the channel dimension and applying shared convolution across the spatial dimension in some computational layers. The proposed method is called CiC. Experimental results on the CIFAR10 dataset, augmented CIFAR10 dataset, and CIFAR100 dataset demonstrate the effectiveness of the proposed CiC method.