InfoTech Assistant: A Multimodal Conversational Agent for InfoTechnology Web Portal Queries

TL;DR

The paper introduces InfoTech Assistant, a multimodal chatbot leveraging LLMs and RAG to provide accurate, context-aware responses for infrastructure technology queries, demonstrating high accuracy and multimodal capabilities.

Contribution

It presents a novel multimodal, domain-specific chatbot integrating web scraping, LLMs, and RAG for infrastructure technology queries, with high accuracy and multimodal response capabilities.

Findings

95% accuracy on domain-specific tasks

High response relevance confirmed by similarity scores

Effective handling of complex multimodal queries

Abstract

This pilot study presents the development of the InfoTech Assistant, a domain-specific, multimodal chatbot engineered to address queries in bridge evaluation and infrastructure technology. By integrating web data scraping, large language models (LLMs), and Retrieval-Augmented Generation (RAG), the InfoTech Assistant provides accurate and contextually relevant responses. Data, including textual descriptions and images, are sourced from publicly available documents on the InfoTechnology website and organized in JSON format to facilitate efficient querying. The architecture of the system includes an HTML-based interface and a Flask back end connected to the Llama 3.1 model via LLM Studio. Evaluation results show approximately 95 percent accuracy on domain-specific tasks, with high similarity scores confirming the quality of response matching. This RAG-enhanced setup enables the InfoTech Assistant to handle complex, multimodal queries, offering both textual and visual information in its responses. The InfoTech Assistant demonstrates strong potential as a dependable tool for infrastructure professionals, delivering high accuracy and relevance in its domain-specific outputs.