Perspective of Software Engineering Researchers on Machine Learning Practices Regarding Research, Review, and Education

TL;DR

This study explores how Software Engineering researchers engage with Machine Learning, revealing diverse practices, common challenges, and gaps in education and methodology that need addressing to advance the field.

Contribution

It provides qualitative insights into SE researchers' practices, challenges, and perspectives on ML, highlighting gaps in current methodologies and educational approaches.

Findings

Diverse practices in data handling, model training, and evaluation.

Less than 20% of literature follows recommended practices like hyperparameter tuning.

Common challenges include data management, model evaluation, and involving human expertise.

Abstract

Context: Machine Learning (ML) significantly impacts Software Engineering (SE), but studies mainly focus on practitioners, neglecting researchers. This overlooks practices and challenges in teaching, researching, or reviewing ML applications in SE. Objective: This study aims to contribute to the knowledge, about the synergy between ML and SE from the perspective of SE researchers, by providing insights into the practices followed when researching, teaching, and reviewing SE studies that apply ML. Method: We analyzed SE researchers familiar with ML or who authored SE articles using ML, along with the articles themselves. We examined practices, SE tasks addressed with ML, challenges faced, and reviewers' and educators' perspectives using grounded theory coding and qualitative analysis. Results: We found diverse practices focusing on data collection, model training, and evaluation. Some recommended practices (e.g., hyperparameter tuning) appeared in less than 20\% of literature. Common challenges involve data handling, model evaluation (incl. non-functional properties), and involving human expertise in evaluation. Hands-on activities are common in education, though traditional methods persist. Conclusion: Despite accepted practices in applying ML to SE, significant gaps remain. By enhancing guidelines, adopting diverse teaching methods, and emphasizing underrepresented practices, the SE community can bridge these gaps and advance the field.