AgentODRL: A Large Language Model-based Multi-agent System for ODRL Generation

TL;DR

This paper presents AgentODRL, a multi-agent system leveraging large language models to automate and improve the accuracy of translating natural language data rights policies into the complex ODRL format, addressing data scarcity and logical complexity.

Contribution

Introduces AgentODRL, a novel multi-agent framework with specialized components and enhanced LLM techniques for high-fidelity ODRL policy generation from natural language.

Findings

Achieved superior ODRL syntax and semantic scores compared to baseline methods.

Demonstrated effectiveness on a new dataset of 770 use cases with varying complexity.

Enhanced policy quality through syntax validation and semantic reflection mechanisms.

Abstract

The Open Digital Rights Language (ODRL) is a pivotal standard for automating data rights management. However, the inherent logical complexity of authorization policies, combined with the scarcity of high-quality "Natural Language-to-ODRL" training datasets, impedes the ability of current methods to efficiently and accurately translate complex rules from natural language into the ODRL format. To address this challenge, this research leverages the potent comprehension and generation capabilities of Large Language Models (LLMs) to achieve both automation and high fidelity in this translation process. We introduce AgentODRL, a multi-agent system based on an Orchestrator-Workers architecture. The architecture consists of specialized Workers, including a Generator for ODRL policy creation, a Decomposer for breaking down complex use cases, and a Rewriter for simplifying nested logical relationships. The Orchestrator agent dynamically coordinates these Workers, assembling an optimal pathway based on the complexity of the input use case. Specifically, we enhance the ODRL Generator by incorporating a validator-based syntax strategy and a semantic reflection mechanism powered by a LoRA-finetuned model, significantly elevating the quality of the generated policies. Extensive experiments were conducted on a newly constructed dataset comprising 770 use cases of varying complexity, all situated within the context of data spaces. The results, evaluated using ODRL syntax and semantic scores, demonstrate that our proposed Orchestrator-Workers system, enhanced with these strategies, achieves superior performance on the ODRL generation task.