Addressing Quality Challenges in Deep Learning: The Role of MLOps and Domain Knowledge

TL;DR

This paper explores how MLOps practices and domain knowledge can improve the quality, transparency, and decision-making process in deep learning system development, emphasizing when to stop optimization efforts.

Contribution

It highlights the importance of MLOps and domain knowledge in addressing quality challenges and provides practical insights from real-world experiences.

Findings

MLOps practices enhance transparency and reproducibility in DL development.

Embedding domain knowledge improves model quality and system integration.

Strategic decision-making on when to stop optimization maximizes system reliability.

Abstract

Deep learning (DL) systems present unique challenges in software engineering, especially concerning quality attributes like correctness and resource efficiency. While DL models excel in specific tasks, engineering DL systems is still essential. The effort, cost, and potential diminishing returns of continual improvements must be carefully evaluated, as software engineers often face the critical decision of when to stop refining a system relative to its quality attributes. This experience paper explores the role of MLOps practices -- such as monitoring and experiment tracking -- in creating transparent and reproducible experimentation environments that enable teams to assess and justify the impact of design decisions on quality attributes. Furthermore, we report on experiences addressing the quality challenges by embedding domain knowledge into the design of a DL model and its integration within a larger system. The findings offer actionable insights into the benefits of domain knowledge and MLOps and the strategic consideration of when to limit further optimizations in DL projects to maximize overall system quality and reliability.