Finding Adversarial Inputs for Heuristics using Multi-level Optimization

TL;DR

MetaOpt is a system that automatically finds adversarial inputs to analyze and improve heuristics across various domains, revealing performance gaps and enabling targeted enhancements.

Contribution

It introduces MetaOpt, a versatile tool that encodes heuristics and their optimal counterparts for scalable analysis of performance gaps and adversarial inputs.

Findings

A traffic heuristic can require 30% more capacity than optimal.

MetaOpt reduced heuristic gap by 12.5× after analysis.

Proved a new lower bound for vector bin packing heuristic.

Abstract

Production systems use heuristics because they are faster or scale better than their optimal counterparts. Yet, practitioners are often unaware of the performance gap between a heuristic and the optimum or between two heuristics in realistic scenarios. We present MetaOpt, a system that helps analyze heuristics. Users specify the heuristic and the optimal (or another heuristic) as input, and MetaOpt automatically encodes these efficiently for a solver to find performance gaps and their corresponding adversarial inputs. Its suite of built-in optimizations helps it scale its analysis to practical problem sizes. To show it is versatile, we used MetaOpt to analyze heuristics from three domains (traffic engineering, vector bin packing, and packet scheduling). We found a production traffic engineering heuristic can require 30% more capacity than the optimal to satisfy realistic demands. Based on the patterns in the adversarial inputs MetaOpt produced, we modified the heuristic to reduce its performance gap by 12.5$\times$. We examined adversarial inputs to a vector bin packing heuristic and proved a new lower bound on its performance.