Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds
Renato L. F. Cunha, Lucas V. Real, Renan Souza, Bruno Silva, Marco A., S. Netto

TL;DR
This paper introduces a Jupyter extension that intelligently migrates notebook cells between local and cloud environments based on user activity, reducing migration time and improving performance for data science tasks.
Contribution
It presents a novel context-aware migration tool that optimizes execution placement of notebook cells, considering user interaction patterns and reducing migration overhead.
Findings
Notebook state reductions of up to 55x achieved.
Migration decisions improved performance by up to 3.25x.
Effective in diverse data science applications like Earth science and machine learning.
Abstract
Interactive computing notebooks, such as Jupyter notebooks, have become a popular tool for developing and improving data-driven models. Such notebooks tend to be executed either in the user's own machine or in a cloud environment, having drawbacks and benefits in both approaches. This paper presents a solution developed as a Jupyter extension that automatically selects which cells, as well as in which scenarios, such cells should be migrated to a more suitable platform for execution. We describe how we reduce the execution state of the notebook to decrease migration time and we explore the knowledge of user interactivity patterns with the notebook to determine which blocks of cells should be migrated. Using notebooks from Earth science (remote sensing), image recognition, and hand written digit identification (machine learning), our experiments show notebook state reductions of up to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
