High-Dimensional Data Processing: Benchmarking Machine Learning and Deep Learning Architectures in Local and Distributed Environments

TL;DR

This paper presents a comprehensive benchmarking of machine learning and deep learning architectures for high-dimensional data processing in local and distributed environments, emphasizing practical workflows and technical implementations.

Contribution

It introduces a detailed methodology for benchmarking ML/DL architectures on high-dimensional data using distributed computing with Apache Spark.

Findings

Effective workflows for high-dimensional data analysis

Performance insights of ML/DL architectures in distributed settings

Implementation guidelines for Spark-based data processing

Abstract

This document reports the sequence of practices and methodologies implemented during the Big Data course. It details the workflow beginning with the processing of the Epsilon dataset through group and individual strategies, followed by text analysis and classification with RestMex and movie feature analysis with IMDb. Finally, it describes the technical implementation of a distributed computing cluster with Apache Spark on Linux using Scala.