Engineering AI Judge Systems

TL;DR

This paper discusses the challenges in developing AI judge systems for FMware, proposes a framework to improve development efficiency and accuracy, and demonstrates its effectiveness through a case study with up to 6.2% accuracy improvement.

Contribution

It introduces a novel framework to address challenges in engineering AI judge systems for FMware, enhancing productivity and judgment accuracy.

Findings

The framework reduces development effort significantly.

Judgment accuracy improves by up to 6.2%.

The approach is validated through a practical case study.

Abstract

AI judge systems are designed to automatically evaluate Foundation Model-powered software (i.e., FMware). Due to the intrinsic dynamic and stochastic nature of FMware, the development of AI judge systems requires a unique engineering life cycle and presents new challenges. In this paper, we discuss the challenges based on our industrial experiences in developing AI judge systems for FMware. These challenges lead to substantial time consumption, cost and inaccurate judgments. We propose a framework that tackles the challenges with the goal of improving the productivity of developing high-quality AI judge systems. Finally, we evaluate our framework with a case study on judging a commit message generation FMware. The accuracy of the judgments made by the AI judge system developed with our framework outperforms those made by the AI judge system that is developed without our framework by up to 6.2%, with a significant reduction in development effort.