TorchTraceAP: A New Benchmark Dataset for Detecting Performance Anti-Patterns in Computer Vision Models

TL;DR

This paper introduces TorchTraceAP, a benchmark dataset and an iterative detection method combining lightweight ML models and LLMs to identify performance anti-patterns in computer vision models' traces, improving detection accuracy.

Contribution

It provides the first benchmark dataset for anti-pattern detection in traces and proposes a novel iterative approach combining ML and LLMs for better detection.

Findings

Our method outperforms unsupervised clustering and rule-based techniques.

It effectively compensates for LLMs' limited context and reasoning.

The dataset includes over 600 diverse PyTorch traces.

Abstract

Identifying and addressing performance anti-patterns in machine learning (ML) models is critical for efficient training and inference, but it typically demands deep expertise spanning system infrastructure, ML models and kernel development. While large tech companies rely on dedicated ML infrastructure engineers to analyze torch traces and benchmarks, such resource-intensive workflows are largely inaccessible to computer vision researchers in general. Among the challenges, pinpointing problematic trace segments within lengthy execution traces remains the most time-consuming task, and is difficult to automate with current ML models, including LLMs. In this work, we present the first benchmark dataset specifically designed to evaluate and improve ML models' ability to detect anti patterns in traces. Our dataset contains over 600 PyTorch traces from diverse computer vision models classification, detection, segmentation, and generation collected across multiple hardware platforms. We also propose a novel iterative approach: a lightweight ML model first detects trace segments with anti patterns, followed by a large language model (LLM) for fine grained classification and targeted feedback. Experimental results demonstrate that our method significantly outperforms unsupervised clustering and rule based statistical techniques for detecting anti pattern regions. Our method also effectively compensates LLM's limited context length and reasoning inefficiencies.