Few-Shot VLM-Based G-Code and HMI Verification in CNC Machining

TL;DR

This paper introduces a few-shot vision-language model approach that jointly verifies G-code and HMI displays in CNC machining, improving error detection over previous LLM-based methods.

Contribution

It presents a novel few-shot VLM-based framework that leverages structured prompts to evaluate both G-code and HMI images for errors, enhancing CNC verification processes.

Findings

Few-shot prompting improves error detection accuracy.

The approach effectively identifies discrepancies between G-code and HMI.

Framework is suitable for manual G-code verification in CNC training.

Abstract

Manual generation of G-code is important for learning the operation of CNC machines. Prior work in G-code verification uses Large-Language Models (LLMs), which primarily examine errors in the written programming. However, CNC machining requires extensive use and knowledge of the Human-Machine Interface (HMI), which displays machine status and errors. LLMs currently lack the capability to leverage knowledge of HMIs due to their inability to access the vision modality. This paper proposes a few-shot VLM-based verification approach that simultaneously evaluates the G-code and the HMI display for errors and safety status. The input dataset includes paired G-code text and associated HMI screenshots from a 15-slant-PRO lathe, including both correct and error-prone cases. To enable few-shot learning, the VLM is provided with a structured JSON schema based on prior heuristic knowledge. After determining the prompts, instances of G-code and HMI that either contain errors or are error free are used as few-shot examples to guide the VLM. The model was then evaluated in comparison to a zero-shot VLM through multiple scenarios of incorrect G-code and HMI errors with respect to per-slot accuracy. The VLM showed that few-shot prompting led to overall enhancement of detecting HMI errors and discrepancies with the G-code for more comprehensive debugging. Therefore, the proposed framework was demonstrated to be suitable for verification of manually generated G-code that is typically developed in CNC training.