DMN-Guided Prompting: A Framework for Controlling LLM Behavior

TL;DR

This paper presents a DMN-guided prompting framework that structures decision logic to improve LLM behavior control, outperforming chain-of-thought prompting and receiving positive user feedback.

Contribution

It introduces a novel framework integrating Decision Model and Notation with prompting to enhance control and effectiveness of large language models.

Findings

Outperformed chain-of-thought prompting in a case study

Received high usefulness ratings from students

Demonstrated effective decision logic management

Abstract

Large Language Models (LLMs) have shown considerable potential in automating decision logic within knowledge-intensive processes. However, their effectiveness largely depends on the strategy and quality of prompting. Since decision logic is typically embedded in prompts, it becomes challenging for end users to modify or refine it. Decision Model and Notation (DMN) offers a standardized graphical approach for defining decision logic in a structured, user-friendly manner. This paper introduces a DMN-guided prompting framework that breaks down complex decision logic into smaller, manageable components, guiding LLMs through structured decision pathways. We implemented the framework in a graduate-level course where students submitted assignments. The assignments and DMN models representing feedback instructions served as inputs to our framework. The instructor evaluated the generated feedback and labeled it for performance assessment. Our approach demonstrated promising results, outperforming chain-of-thought (CoT) prompting in our case study. Students also responded positively to the generated feedback, reporting high levels of perceived usefulness in a survey based on the Technology Acceptance Model.