Evaluating perturbation robustness of generative systems that use COBOL code inputs

TL;DR

This paper introduces a framework to evaluate and visualize the robustness of LLM-based systems processing COBOL code, addressing sensitivity issues in critical legacy applications and providing tools for debugging and improvement.

Contribution

It develops perturbation methods for COBOL code, creates a dataset with variants, and provides visualization dashboards to analyze and enhance system robustness.

Findings

Robustness metrics reveal sensitivity to input variations

Visualization tools assist in debugging and understanding system behavior

Framework applicable to other code-related tasks

Abstract

Systems incorporating large language models (LLMs) as a component are known to be sensitive (i.e., non-robust) to minor input variations that do not change the meaning of the input; such sensitivity may reduce the system's usefulness. Here, we present a framework to evaluate robustness of systems using COBOL code as input; our application is translation between COBOL and Java programming languages, but the approach extends to other tasks such as code generation or explanation. Targeting robustness of systems with COBOL as input is essential yet challenging. Many business-critical applications are written in COBOL, yet these are typically proprietary legacy applications and their code is unavailable to LLMs for training. We develop a library of COBOL paragraph and full-program perturbation methods, and create variant-expanded versions of a benchmark dataset of examples for a specific task. The robustness of the LLM-based system is evaluated by measuring changes in values of individual and aggregate metrics calculated on the system's outputs. Finally, we present a series of dynamic table and chart visualization dashboards that assist in debugging the system's outputs, and monitoring and understanding root causes of the system's sensitivity to input variation. These tools can be further used to improve the system by, for instance, indicating variations that should be handled by pre-processing steps.