Developing an AI Assistant for Knowledge Management and Workforce Training in State DOTs

TL;DR

This paper introduces a multi-agent Retrieval-Augmented Generation framework utilizing large language models and visual understanding to enhance knowledge management and training in state transportation agencies, addressing information fragmentation and retrieval challenges.

Contribution

It presents a novel multi-agent RAG system with figure understanding capabilities, improving knowledge retrieval and decision support for state DOTs.

Findings

Enhanced retrieval accuracy with multi-agent architecture

Effective integration of visual and textual data for knowledge retrieval

Improved decision support in transportation agency workflows

Abstract

Effective knowledge management is critical for preserving institutional expertise and improving the efficiency of workforce training in state transportation agencies. Traditional approaches, such as static documentation, classroom-based instruction, and informal mentorship, often lead to fragmented knowledge transfer, inefficiencies, and the gradual loss of expertise as senior engineers retire. Moreover, given the enormous volume of technical manuals, guidelines, and research reports maintained by these agencies, it is increasingly challenging for engineers to locate relevant information quickly and accurately when solving field problems or preparing for training tasks. These limitations hinder timely decision-making and create steep learning curves for new personnel in maintenance and construction operations. To address these challenges, this paper proposes a Retrieval-Augmented Generation (RAG) framework with a multi-agent architecture to support knowledge management and decision making. The system integrates structured document retrieval with real-time, context-aware response generation powered by a large language model (LLM). Unlike conventional single-pass RAG systems, the proposed framework employs multiple specialized agents for retrieval, answer generation, evaluation, and query refinement, which enables iterative improvement and quality control. In addition, the system incorporates an open-weight vision-language model to convert technical figures into semantic textual representations, which allows figure-based knowledge to be indexed and retrieved alongside text. Retrieved text and figure-based context are then provided to an open-weight large language model, which generates the final responses grounded in the retrieved evidence.