Towards Automatic Translation of Machine Learning Visual Insights to Analytical Assertions

TL;DR

This paper envisions creating an automated tool that translates visual insights from ML visualizations into Python assertions, aiming to improve verification processes in ML development using AI models and large datasets.

Contribution

It proposes a novel approach to automate the translation of ML visualization properties into assertions, leveraging a large dataset and comparing AI models for effective translation.

Findings

Comparison of AI models for translation effectiveness

Development of a taxonomy for visualization-assertion pairs

Plans to extend dataset with Kaggle data

Abstract

We present our vision for developing an automated tool capable of translating visual properties observed in Machine Learning (ML) visualisations into Python assertions. The tool aims to streamline the process of manually verifying these visualisations in the ML development cycle, which is critical as real-world data and assumptions often change post-deployment. In a prior study, we mined $54,070$ Jupyter notebooks from Github and created a catalogue of $269$ semantically related visualisation-assertion (VA) pairs. Building on this catalogue, we propose to build a taxonomy that organises the VA pairs based on ML verification tasks. The input feature space comprises of a rich source of information mined from the Jupyter notebooks -- visualisations, Python source code, and associated markdown text. The effectiveness of various AI models, including traditional NLP4Code models and modern Large Language Models, will be compared using established machine translation metrics and evaluated through a qualitative study with human participants. The paper also plans to address the challenge of extending the existing VA pair dataset with additional pairs from Kaggle and to compare the tool's effectiveness with commercial generative AI models like ChatGPT. This research not only contributes to the field of ML system validation but also explores novel ways to leverage AI for automating and enhancing software engineering practices in ML.