mPyPl: Python Monadic Pipeline Library for Complex Functional Data Processing

TL;DR

mPyPl is a Python library that simplifies complex data processing by enabling functional pipelines on lazy multi-field data streams, inspired by monadic operations, and applied to deep learning event detection in videos.

Contribution

The paper introduces mPyPl, a novel Python library that implements monadic-like operations on data streams for streamlined complex data processing tasks.

Findings

Effective data stream enrichment and feature extraction

Flexible evaluation strategies balancing memory and performance

Successful application in deep learning event detection

Abstract

In this paper, we present a new Python library called mPyPl, which is intended to simplify complex data processing tasks using functional approach. This library defines operations on lazy data streams of named dictionaries represented as generators (so-called multi-field datastreams), and allows enriching those data streams with more 'fields' in the process of data preparation and feature extraction. Thus, most data preparation tasks can be expressed in the form of neat linear 'pipeline', similar in syntax to UNIX pipes, or |> functional composition operator in F#. We define basic operations on multi-field data streams, which resemble classical monadic operations, and show similarity of the proposed approach to monads in functional programming. We also show how the library was used in complex deep learning tasks of event detection in video, and discuss different evaluation strategies that allow for different compromises in terms of memory and performance.