Dense Optimizer : An Information Entropy-Guided Structural Search Method for Dense-like Neural Network Design

TL;DR

The paper introduces Dense Optimizer, an automatic architecture search method guided by information entropy, which efficiently designs dense-like neural networks with improved accuracy on vision benchmarks.

Contribution

It proposes a novel entropy-guided search framework and a branch-and-bound algorithm for automatic dense network design, reducing manual effort and improving performance.

Findings

Achieved 84.3% top-1 accuracy on CIFAR-100, outperforming the original DenseNet.

Completed high-quality architecture search in only 4 hours on a single CPU.

Validated the method's effectiveness across multiple computer vision datasets.

Abstract

Dense Convolutional Network has been continuously refined to adopt a highly efficient and compact architecture, owing to its lightweight and efficient structure. However, the current Dense-like architectures are mainly designed manually, it becomes increasingly difficult to adjust the channels and reuse level based on past experience. As such, we propose an architecture search method called Dense Optimizer that can search high-performance dense-like network automatically. In Dense Optimizer, we view the dense network as a hierarchical information system, maximize the network's information entropy while constraining the distribution of the entropy across each stage via a power law, thereby constructing an optimization problem. We also propose a branch-and-bound optimization algorithm, tightly integrates power-law principle with search space scaling to solve the optimization problem efficiently. The superiority of Dense Optimizer has been validated on different computer vision benchmark datasets. Specifically, Dense Optimizer completes high-quality search but only costs 4 hours with one CPU. Our searched model DenseNet-OPT achieved a top 1 accuracy of 84.3% on CIFAR-100, which is 5.97% higher than the original one.