On Using Information Retrieval to Recommend Machine Learning Good Practices for Software Engineers

TL;DR

This paper introduces Idaka, a tool that uses information retrieval and large language models to recommend machine learning best practices to software engineers, aiming to improve ML system performance and usability.

Contribution

The paper presents a novel recommender system combining IR and LLM approaches to help users find relevant ML practices from various sources.

Findings

Implemented Idaka with BM25 and Alpaca models

Provides a platform for comparative evaluation of retrieval methods

Available publicly for further research and development

Abstract

Machine learning (ML) is nowadays widely used for different purposes and in several disciplines. From self-driving cars to automated medical diagnosis, machine learning models extensively support users' daily activities, and software engineering tasks are no exception. Not embracing good ML practices may lead to pitfalls that hinder the performance of an ML system and potentially lead to unexpected results. Despite the existence of documentation and literature about ML best practices, many non-ML experts turn towards gray literature like blogs and Q&A systems when looking for help and guidance when implementing ML systems. To better aid users in distilling relevant knowledge from such sources, we propose a recommender system that recommends ML practices based on the user's context. As a first step in creating a recommender system for machine learning practices, we implemented Idaka. A tool that provides two different approaches for retrieving/generating ML best practices: i) an information retrieval (IR) engine and ii) a large language model. The IR-engine uses BM25 as the algorithm for retrieving the practices, and a large language model, in our case Alpaca. The platform has been designed to allow comparative studies of best practices retrieval tools. Idaka is publicly available at GitHub: https://bit.ly/idaka. Video: https://youtu.be/cEb-AhIPxnM.