An Efficient Hardware-Oriented Dropout Algorithm

TL;DR

This paper introduces a hardware-oriented dropout algorithm optimized for FPGA implementation, reducing resource usage while maintaining performance, thus enabling efficient training of deep neural networks in embedded systems.

Contribution

The paper presents a novel dropout algorithm tailored for FPGA hardware, addressing memory and power constraints in embedded neural network training.

Findings

Performance identical to conventional dropout

Significant resource reduction in FPGA synthesis

Efficient utilization of FPGA characteristics

Abstract

This paper proposes a hardware-oriented dropout algorithm, which is efficient for field programmable gate array (FPGA) implementation. In deep neural networks (DNNs), overfitting occurs when networks are overtrained and adapt too well to training data. Consequently, they fail in predicting unseen data used as test data. Dropout is a common technique that is often applied in DNNs to overcome this problem. In general, implementing such training algorithms of DNNs in embedded systems is difficult due to power and memory constraints. Training DNNs is power-, time-, and memory- intensive; however, embedded systems require low power consumption and real-time processing. An FPGA is suitable for embedded systems for its parallel processing characteristic and low operating power; however, due to its limited memory and different architecture, it is difficult to apply general neural network algorithms. Therefore, we propose a hardware-oriented dropout algorithm that can effectively utilize the characteristics of an FPGA with less memory required. Software program verification demonstrates that the performance of the proposed method is identical to that of conventional dropout, and hardware synthesis demonstrates that it results in significant resource reduction.