TL;DR
This paper introduces a progressive stochastic binarization method for deep networks that enables efficient, adaptive, and unbiased low-precision inference, achieving accuracy close to full-precision models with reduced computational costs.
Contribution
It presents a novel progressive stochastic binarization scheme allowing adaptive accuracy control and efficient inference, outperforming previous binarization methods in flexibility and cost-effectiveness.
Findings
Achieves near-original accuracy on ImageNet with low representational costs.
Reduces inference costs by up to 33% through adaptive sampling.
Compatible with pretrained networks, including pruned models.
Abstract
A plethora of recent research has focused on improving the memory footprint and inference speed of deep networks by reducing the complexity of (i) numerical representations (for example, by deterministic or stochastic quantization) and (ii) arithmetic operations (for example, by binarization of weights). We propose a stochastic binarization scheme for deep networks that allows for efficient inference on hardware by restricting itself to additions of small integers and fixed shifts. Unlike previous approaches, the underlying randomized approximation is progressive, thus permitting an adaptive control of the accuracy of each operation at run-time. In a low-precision setting, we match the accuracy of previous binarized approaches. Our representation is unbiased - it approaches continuous computation with increasing sample size. In a high-precision regime, the computational costs are…
Click any figure to enlarge with its caption.
_graphics_i0.png)
Figure 1_graphics_i0_l0_mean_mean.png)
Figure 2_graphics_i0_l48_entropy.png)
Figure 3_graphics_i0_l48_mask.png)
Figure 4_graphics_i0_l48_mean_mean.png)
Figure 5_illustrations_AccumulationDataFlow.png)
Figure 6_illustrations_AccumulationDataFlow2.png)
Figure 7_illustrations_Accumulator.png)
Figure 8_illustrations_StochasticMultiplier.png)
Figure 9_illustrations_StochasticMultiplier2.png)
Figure 10_illustrations_docProps_thumbnail.jpeg)
Figure 11_illustrations_ppt_media_image1.png)
Figure 12_illustrations_ppt_media_image10.png)
Figure 13_illustrations_ppt_media_image11.png)
Figure 14_illustrations_ppt_media_image13.png)
Figure 15_illustrations_ppt_media_image15.png)
Figure 16_illustrations_ppt_media_image16.png)
Figure 17_illustrations_ppt_media_image17.png)
Figure 18_illustrations_ppt_media_image18.png)
Figure 19_illustrations_ppt_media_image19.png)
Figure 20_illustrations_ppt_media_image2.png)
Figure 21_illustrations_ppt_media_image20.png)
Figure 22_illustrations_ppt_media_image21.png)
Figure 23_illustrations_ppt_media_image22.png)
Figure 24_illustrations_ppt_media_image23.png)
Figure 25_illustrations_ppt_media_image25.png)
Figure 26_illustrations_ppt_media_image26.png)
Figure 27_illustrations_ppt_media_image27.png)
Figure 28_illustrations_ppt_media_image28.png)
Figure 29_illustrations_ppt_media_image29.png)
Figure 30_illustrations_ppt_media_image3.png)
Figure 31_illustrations_ppt_media_image30.png)
Figure 32_illustrations_ppt_media_image31.png)
Figure 33_illustrations_ppt_media_image310.png)
Figure 34_illustrations_ppt_media_image33.png)
Figure 35_illustrations_ppt_media_image36.png)
Figure 36_illustrations_ppt_media_image370.png)
Figure 37_illustrations_ppt_media_image380.png)
Figure 38_illustrations_ppt_media_image39.png)
Figure 39_illustrations_ppt_media_image390.png)
Figure 40Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsSPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings
Progressive Stochastic Binarization of Deep Networks
David Hartmann
Institute of Computer Science
Johannes Gutenberg-University of Mainz
Staudingerweg 9, 55128 Mainz, Germany
&Michael Wand
Institute of Computer Science
Johannes Gutenberg-University of Mainz
Staudingerweg 9, 55128 Mainz, Germany
\ExecuteMetaData
[sections/structure.tex]
Supplementary Material for:
Progressive Stochastic Binarization of Deep Networks
\ExecuteMetaData
[sections_supplementary/structure.tex]
