You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack

TL;DR

This paper presents a serverless, open-source data pipeline template for machine learning at a reasonable scale, enabling industry practitioners to leverage recommender system research without extensive infrastructure.

Contribution

It introduces a practical, scalable data stack that simplifies pipeline setup using serverless and open-source tools, addressing industry challenges with immature data pipelines.

Findings

Modern open source tools can process terabytes of data efficiently.

Serverless paradigms reduce infrastructure complexity.

The proposed template improves data pipeline reliability and scalability.

Abstract

We argue that immature data pipelines are preventing a large portion of industry practitioners from leveraging the latest research on recommender systems. We propose our template data stack for machine learning at "reasonable scale", and show how many challenges are solved by embracing a serverless paradigm. Leveraging our experience, we detail how modern open source can provide a pipeline processing terabytes of data with limited infrastructure work.