Hybrid Context Retrieval Augmented Generation Pipeline: LLM-Augmented Knowledge Graphs and Vector Database for Accreditation Reporting Assistance

TL;DR

This paper presents a hybrid retrieval-augmented generation pipeline combining knowledge graphs and vector databases to assist higher education institutions in accreditation reporting, improving relevance and correctness of generated content.

Contribution

It introduces a novel hybrid context retrieval pipeline utilizing knowledge graphs and vector databases, enhanced by LLM augmentation, for accreditation report assistance.

Findings

Optimal performance on answer relevancy metrics

High answer correctness observed in evaluations

Effective integration of knowledge graphs and vector databases

Abstract

In higher education, accreditation is a quality assurance process, where an institution demonstrates a commitment to delivering high quality programs and services to their students. For business schools nationally and internationally the Association to Advance Collegiate Schools of Business (AACSB) accreditation is the gold standard. For a business school to receive and subsequently maintain accreditation, the school must undertake a rigorous, time consuming reporting and peer review process, to demonstrate alignment with the AACSB Standards. For this project we create a hybrid context retrieval augmented generation pipeline that can assist in the documentation alignment and reporting process necessary for accreditation. We implement both a vector database and knowledge graph, as knowledge stores containing both institutional data and AACSB Standard data. The output of the pipeline can be used by institution stakeholders to build their accreditation report, dually grounded by the context from the knowledge stores. To develop our knowledge graphs we utilized both a manual construction process as well as an LLM Augmented Knowledge Graph approach. We evaluated the pipeline using the RAGAs framework and observed optimal performance on answer relevancy and answer correctness metrics.