An Architecture for Probabilistic Concept-Based Information Retrieval

TL;DR

This paper proposes a probabilistic architecture for concept-based information retrieval that automates knowledge acquisition, demonstrating its feasibility and advantages through experiments on terrorism-related document retrieval.

Contribution

It introduces a novel architecture utilizing probabilistic networks to automate concept knowledge base construction for improved information retrieval.

Findings

Architecture is feasible for real-world data

Concept-based methods outperform traditional techniques

Automated knowledge acquisition reduces manual effort

Abstract

While concept-based methods for information retrieval can provide improved performance over more conventional techniques, they require large amounts of effort to acquire the concepts and their qualitative and quantitative relationships. This paper discusses an architecture for probabilistic concept-based information retrieval which addresses the knowledge acquisition problem. The architecture makes use of the probabilistic networks technology for representing and reasoning about concepts and includes a knowledge acquisition component which partially automates the construction of concept knowledge bases from data. We describe two experiments that apply the architecture to the task of retrieving documents about terrorism from a set of documents from the Reuters news service. The experiments provide positive evidence that the architecture design is feasible and that there are advantages to concept-based methods.