Performance Analysis of Open Source Machine Learning Frameworks for Various Parameters in Single-Threaded and Multi-Threaded Modes

TL;DR

This paper compares popular open source machine learning frameworks, analyzing their performance on MNIST data across different parameters and modes, highlighting optimization opportunities for image recognition tasks.

Contribution

It provides a comparative analysis of TensorFlow, Deep Learning4j, and H2O, including performance testing and parameter impact on neural network efficiency.

Findings

Parameter selection can drastically increase runtime without accuracy gains

H2O framework performs well on CPU and GPU for MNIST

Optimization of parameters is crucial for efficient image recognition

Abstract

The basic features of some of the most versatile and popular open source frameworks for machine learning (TensorFlow, Deep Learning4j, and H2O) are considered and compared. Their comparative analysis was performed and conclusions were made as to the advantages and disadvantages of these platforms. The performance tests for the de facto standard MNIST data set were carried out on H2O framework for deep learning algorithms designed for CPU and GPU platforms for single-threaded and multithreaded modes of operation Also, we present the results of testing neural networks architectures on H2O platform for various activation functions, stopping metrics, and other parameters of machine learning algorithm. It was demonstrated for the use case of MNIST database of handwritten digits in single-threaded mode that blind selection of these parameters can hugely increase (by 2-3 orders) the runtime without the significant increase of precision. This result can have crucial influence for optimization of available and new machine learning methods, especially for image recognition problems.