GEMS: Generative Expert Metric System through Iterative Prompt Priming

TL;DR

This paper introduces GEMS, a prompt-engineering framework leveraging generative models to create context-aware metrics from theories, aiding complex decision-making across domains, especially in software communities.

Contribution

It presents a novel framework that uses iterative prompt priming with generative models to transform theories into practical, context-specific metrics for complex data analysis.

Findings

Demonstrates effective extraction and summarization of theories.

Shows capability to perform basic reasoning with generative models.

Applicable across multiple domains beyond software communities.

Abstract

Across domains, metrics and measurements are fundamental to identifying challenges, informing decisions, and resolving conflicts. Despite the abundance of data available in this information age, not only can it be challenging for a single expert to work across multi-disciplinary data, but non-experts can also find it unintuitive to create effective measures or transform theories into context-specific metrics that are chosen appropriately. This technical report addresses this challenge by examining software communities within large software corporations, where different measures are used as proxies to locate counterparts within the organization to transfer tacit knowledge. We propose a prompt-engineering framework inspired by neural activities, demonstrating that generative models can extract and summarize theories and perform basic reasoning, thereby transforming concepts into context-aware metrics to support software communities given software repository data. While this research zoomed in on software communities, we believe the framework's applicability extends across various fields, showcasing expert-theory-inspired metrics that aid in triaging complex challenges.