How far can we go without convolution: Improving fully-connected networks

TL;DR

This paper explores methods to enhance fully connected neural networks, achieving high accuracy on CIFAR-10 without convolutional layers by using linear bottlenecks and autoencoder pre-training.

Contribution

It introduces two effective techniques—linear bottleneck layers and bias-free autoencoder pre-training—to significantly improve fully connected network performance.

Findings

Achieved approximately 70% accuracy on CIFAR-10 without convolution.

Enhanced accuracy to 78% with data augmentation, nearing convolutional network performance.

Linked improvements to better gradient flow and reduced sparsity in networks.

Abstract

We propose ways to improve the performance of fully connected networks. We found that two approaches in particular have a strong effect on performance: linear bottleneck layers and unsupervised pre-training using autoencoders without hidden unit biases. We show how both approaches can be related to improving gradient flow and reducing sparsity in the network. We show that a fully connected network can yield approximately 70% classification accuracy on the permutation-invariant CIFAR-10 task, which is much higher than the current state-of-the-art. By adding deformations to the training data, the fully connected network achieves 78% accuracy, which is just 10% short of a decent convolutional network.