Optimistic Estimate Uncovers the Potential of Nonlinear Models
Yaoyu Zhang, Zhongwang Zhang, Leyang Zhang, Zhiwei Bai, Tao Luo,, Zhi-Qin John Xu

TL;DR
This paper introduces an optimistic estimate to evaluate the best possible fitting performance of nonlinear models, providing insights into their overparameterization capabilities and guiding architecture design.
Contribution
It presents a novel optimistic estimate for the minimal sample size needed for nonlinear models, revealing properties of DNNs and informing architecture choices.
Findings
Optimistic sample size estimates predict fitability of targets.
DNNs exhibit free expressiveness in width and costly expressiveness in connections.
The framework enhances understanding of nonlinear models' overparameterization potential.
Abstract
We propose an optimistic estimate to evaluate the best possible fitting performance of nonlinear models. It yields an optimistic sample size that quantifies the smallest possible sample size to fit/recover a target function using a nonlinear model. We estimate the optimistic sample sizes for matrix factorization models, deep models, and deep neural networks (DNNs) with fully-connected or convolutional architecture. For each nonlinear model, our estimates predict a specific subset of targets that can be fitted at overparameterization, which are confirmed by our experiments. Our optimistic estimate reveals two special properties of the DNN models -- free expressiveness in width and costly expressiveness in connection. These properties suggest the following architecture design principles of DNNs: (i) feel free to add neurons/kernels; (ii) restrain from connecting neurons. Overall, our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Model Reduction and Neural Networks · Stochastic Gradient Optimization Techniques
