KaPQA: Knowledge-Augmented Product Question-Answering

TL;DR

This paper introduces new domain-specific QA datasets for Adobe products and proposes a knowledge-driven RAG-QA framework, showing modest improvements in product question-answering performance.

Contribution

The paper provides the first product-specific QA datasets for Adobe Acrobat and Photoshop and develops a novel knowledge-augmented RAG-QA framework for improved domain-specific QA.

Findings

Knowledge induction via query reformulation improves retrieval and generation.

The datasets reveal the challenge of domain-specific QA performance.

Modest performance gains highlight the difficulty of the task.

Abstract

Question-answering for domain-specific applications has recently attracted much interest due to the latest advancements in large language models (LLMs). However, accurately assessing the performance of these applications remains a challenge, mainly due to the lack of suitable benchmarks that effectively simulate real-world scenarios. To address this challenge, we introduce two product question-answering (QA) datasets focused on Adobe Acrobat and Photoshop products to help evaluate the performance of existing models on domain-specific product QA tasks. Additionally, we propose a novel knowledge-driven RAG-QA framework to enhance the performance of the models in the product QA task. Our experiments demonstrated that inducing domain knowledge through query reformulation allowed for increased retrieval and generative performance when compared to standard RAG-QA methods. This improvement, however, is slight, and thus illustrates the challenge posed by the datasets introduced.