Statistical Software Engineering with Tuned Variables

TL;DR

This paper proposes managing AI system components as governed, tunable variables within a versioned program space, emphasizing statistical evaluation and adaptive governance to handle evolving environments.

Contribution

It introduces a framework treating system choices as tunable variables governed by statistical criteria, extending software engineering to AI-enabled systems under change.

Findings

System choices are better managed as governed, tunable variables.

Statistical evaluation enables adaptive promotion of system configurations.

Framework supports evolving environments and evaluation sets.

Abstract

The maintained artifact in an AI-enabled system is not code plus settings, but a versioned governed program space: domains, structural constraints, eligibility, evaluation assets, and a statistical release gate. AI-enabled systems operate under changing world conditions: provider models and APIs change, input distributions drift, evaluation sets age, and objectives such as quality, cost, latency, and safety are renegotiated over time. In practice, teams often respond through ad hoc changes to model choice, retrieval policy, prompt structure, and operational thresholds. Fixed-assignment reasoning is therefore insufficient: a chosen assignment is valid only relative to an environment, evaluation set, and policy state. We argue that such choices should be treated as tuned variables: program variables maintained under governance as environments and evaluation sets evolve. Building on SE4AI work and our prior work on governed tuning, this paper positions the governed space as the software-engineering object. Here, statistical means that promotion relies on sampled evaluation sets, estimated evidence, effect-size margins, and confidence/risk thresholds.