Feature Engineering for Agents: An Adaptive Cognitive Architecture for Interpretable ML Monitoring

TL;DR

This paper introduces a cognitive architecture that applies feature engineering principles to LLM-based ML monitoring, significantly improving interpretability and decision support in production environments.

Contribution

It presents a novel Decision Procedure module that refactors, breaks down, and compiles monitoring data for enhanced interpretability and robustness.

Findings

Higher accuracy in monitoring across multiple domains

Improved interpretability of ML monitoring outputs

Reduced reliance on LLM planning, increasing robustness

Abstract

Monitoring Machine Learning (ML) models in production environments is crucial, yet traditional approaches often yield verbose, low-interpretability outputs that hinder effective decision-making. We propose a cognitive architecture for ML monitoring that applies feature engineering principles to agents based on Large Language Models (LLMs), significantly enhancing the interpretability of monitoring outputs. Central to our approach is a Decision Procedure module that simulates feature engineering through three key steps: Refactor, Break Down, and Compile. The Refactor step improves data representation to better capture feature semantics, allowing the LLM to focus on salient aspects of the monitoring data while reducing noise and irrelevant information. Break Down decomposes complex information for detailed analysis, and Compile integrates sub-insights into clear, interpretable outputs. This process leads to a more deterministic planning approach, reducing dependence on LLM-generated planning, which can sometimes be inconsistent and overly general. The combination of feature engineering-driven planning and selective LLM utilization results in a robust decision support system, capable of providing highly interpretable and actionable insights. Experiments using multiple LLMs demonstrate the efficacy of our approach, achieving significantly higher accuracy compared to various baselines across several domains.