RoD-TAL: A Benchmark for Answering Questions in Romanian Driving License Exams

TL;DR

This paper introduces RoD-TAL, a multimodal dataset for Romanian driving license exam questions, and evaluates LLMs and VLMs in legal question-answering tasks, showing promising results with domain-specific fine-tuning and reasoning techniques.

Contribution

The paper presents RoD-TAL, a new multimodal dataset for Romanian driving law questions, and assesses advanced AI models' capabilities in legal reasoning and question-answering tasks.

Findings

Domain-specific fine-tuning improves retrieval performance.

Chain-of-thought prompting enhances question-answering accuracy.

Models can surpass minimum passing grades in driving exams.

Abstract

The intersection of AI and legal systems presents a growing need for tools that support legal education, particularly in under-resourced languages such as Romanian. In this work, we aim to evaluate the capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs) in understanding and reasoning about the Romanian driving law through textual and visual question-answering tasks. To facilitate this, we introduce RoD-TAL, a novel multimodal dataset comprising Romanian driving test questions, text-based and image-based, along with annotated legal references and explanations written by human experts. We implement and assess retrieval-augmented generation (RAG) pipelines, dense retrievers, and reasoning-optimized models across tasks, including Information Retrieval (IR), Question Answering (QA), Visual IR, and Visual QA. Our experiments demonstrate that domain-specific fine-tuning significantly enhances retrieval performance. At the same time, chain-of-thought prompting and specialized reasoning models improve QA accuracy, surpassing the minimum passing grades required for driving exams. We highlight the potential and limitations of applying LLMs and VLMs to legal education. We release the code and resources through the GitHub repository.