Using LLMs and Essence to Support Software Practice Adoption

TL;DR

This paper presents a specialized chatbot integrating Essence and large language models to support software engineering practice adoption, demonstrating improved relevance and response quality over general-purpose LLMs.

Contribution

It introduces a retrieval-augmented LLM system tailored for Essence, enhancing domain-specific support in software engineering practices.

Findings

The system outperforms baseline LLMs in relevance and response quality.

Retrieval-augmented generation improves domain-specific knowledge access.

Potential to enhance software process management and adoption.

Abstract

Recent advancements in natural language processing (NLP) have enabled the development of automated tools that support various domains, including software engineering. However, while NLP and artificial intelligence (AI) research has extensively focused on tasks such as code generation, less attention has been given to automating support for the adoption of best practices, the evolution of ways of working, and the monitoring of process health. This study addresses this gap by exploring the integration of Essence, a standard and thinking framework for managing software engineering practices, with large language models (LLMs). To this end, a specialised chatbot was developed to assist students and professionals in understanding and applying Essence. The chatbot employs a retrieval-augmented generation (RAG) system to retrieve relevant contextual information from a curated knowledge base. Four different LLMs were used to create multiple chatbot configurations, each evaluated both as a base model and augmented with the RAG system. The system performance was evaluated through both the relevance of retrieved context and the quality of generated responses. Comparative analysis against the general-purpose LLMs demonstrated that the proposed system consistently outperforms its baseline counterpart in domain-specific tasks. By facilitating access to structured software engineering knowledge, this work contributes to bridging the gap between theoretical frameworks and practical application, potentially improving process management and the adoption of software development practices. While further validation through user studies is required, these findings highlight the potential of LLM-based automation to enhance learning and decision-making in software engineering.