Bio-Eng-LMM AI Assist chatbot: A Comprehensive Tool for Research and Education

TL;DR

Bio-Eng-LMM AI chatbot is a versatile, open-source platform that combines retrieval-augmented generation, image processing, and web search to support research and education through interactive text and voice responses.

Contribution

It introduces a comprehensive AI chatbot integrating RAG, image understanding, and web search, enhancing research and educational interactions with LLMs.

Findings

Effective retrieval-augmented responses from multiple sources

Supports image generation and understanding within chatbot interactions

Provides coherent multi-modal communication in research and education

Abstract

This article introduces Bio-Eng-LMM AI chatbot, a versatile platform designed to enhance user interaction for educational and research purposes. Leveraging cutting-edge open-source Large Language Models (LLMs), Bio-Eng-LMM operates as a sophisticated AI assistant, exploiting the capabilities of traditional models like ChatGPT. Central to Bio-Eng-LMM is its implementation of Retrieval Augmented Generation (RAG) through three primary methods: integration of preprocessed documents, real-time processing of user-uploaded files, and information retrieval from any specified website. Additionally, the chatbot incorporates image generation via a Stable Diffusion Model (SDM), image understanding and response generation through LLAVA, and search functionality on the internet powered by secure search engine such as DuckDuckGo. To provide comprehensive support, Bio-Eng-LMM offers text summarization, website content summarization, and both text and voice interaction. The chatbot maintains session memory to ensure contextually relevant and coherent responses. This integrated platform builds upon the strengths of RAG-GPT and Web-Based RAG Query (WBRQ) where the system fetches relevant information directly from the web to enhance the LLMs response generation.