GRAFITE: Generative Regression Analysis Framework for Issue Tracking and Evaluation

TL;DR

GRAFITE is a platform for continuous evaluation of large language models, addressing data contamination issues and enabling regression detection through user feedback and QA testing.

Contribution

It introduces a comprehensive system for ongoing LLM evaluation using user feedback, QA tests, and model comparison to detect performance regressions over time.

Findings

Enables side-by-side comparison of multiple LLMs

Detects performance regressions across model releases

Utilizes user feedback for issue repository

Abstract

Large language models (LLMs) are largely motivated by their performance on popular topics and benchmarks at the time of their release. However, over time, contamination occurs due to significant exposure of benchmark data during training. This poses a risk of model performance inflation if testing is not carefully executed. To address this challenge, we present GRAFITE, a continuous LLM evaluation platform through a comprehensive system for maintaining and evaluating model issues. Our approach enables building a repository of model problems based on user feedback over time and offers a pipeline for assessing LLMs against these issues through quality assurance (QA) tests using LLM-as-a-judge. The platform enables side-by-side comparison of multiple models, facilitating regression detection across different releases. The platform is available at https://github.com/IBM/grafite. The demo video is available at www.youtube.com/watch?v=XFZyoleN56k.