Enhancing Semantic Document Retrieval- Employing Group Steiner Tree Algorithm with Domain Knowledge Enrichment

TL;DR

This paper introduces a novel semantic document retrieval algorithm that integrates domain knowledge using a Group Steiner Tree approach, significantly improving retrieval precision and accuracy in real-world scenarios.

Contribution

The paper presents a new algorithm combining Group Steiner Tree with domain knowledge for enhanced semantic document retrieval, validated through real-world data and expert evaluation.

Findings

Achieved 90% precision in retrieval tasks.

Attained 82% accuracy in system performance.

Demonstrated substantial improvements over baseline systems.

Abstract

Retrieving pertinent documents from various data sources with diverse characteristics poses a significant challenge for Document Retrieval Systems. The complexity of this challenge is further compounded when accounting for the semantic relationship between data and domain knowledge. While existing retrieval systems using semantics (usually represented as Knowledge Graphs created from open-access resources and generic domain knowledge) hold promise in delivering relevant outcomes, their precision may be compromised due to the absence of domain-specific information and reliance on outdated knowledge sources. In this research, the primary focus is on two key contributions- a) the development of a versatile algorithm- 'Semantic-based Concept Retrieval using Group Steiner Tree' that incorporates domain information to enhance semantic-aware knowledge representation and data access, and b) the practical implementation of the proposed algorithm within a document retrieval system using real-world data. To assess the effectiveness of the SemDR system, research work conducts performance evaluations using a benchmark consisting of 170 real-world search queries. Rigorous evaluation and verification by domain experts are conducted to ensure the validity and accuracy of the results. The experimental findings demonstrate substantial advancements when compared to the baseline systems, with precision and accuracy achieving levels of 90% and 82% respectively, signifying promising improvements.

Tables

Table 1. TABLE I: Test-cases : The table elaborates on various use cases and the approaches used by the proposed algorithm

No.	Use-case	Relevant Concepts Identified by Proposed Algorithm
1.	No anchor concept identified	No relevant concepts retrieved. In Figure 2 subfigure (C), let’s consider a different search string, namely ‘Silk Import’. It is observed that only one concept is identified as an Anchor concept-‘Silk’, which is most dominantly related to the search string with high semantic proximity score. Subsequently, files associated with the identified Anchor concept-‘Silk’, are retrieved from the data repository.
2.	Single anchor concept identified	Documents related to single concept retrieved. In the context of Figure 2 subfigure (C), let’s examine the search string ‘Education Facilities’. Upon analysis, it is determined that this search string lacks any meaningful semantic relativeness with the domain or the existing Anchor concepts. As a result, the system fails to identify any Anchor concept corresponding to the search string ‘Education Facilities’. Consequently, there are no relevant Anchor concepts available to guide the retrieval process.
3.	Multiple anchor concepts are identified which are directly linked	The files associated with these identified Anchor concepts are fetched, and the files common among all the concepts are given the most importance. In Figure 2 subfigure (c), we illustrate a scenario involving a search string, denoted as ‘Chilly Export’. This search string is mapped to two Anchor concepts: Chilly’ and ‘Chilly Export’. It is important to note that a direct path exists between these two Anchor concepts. To accomplish the retrieval process, the files associated with these identified Anchor concepts are fetched from the data repository. These retrieved files are then subjected to a comparison process. Specifically, the files common to both Chilly’ and Chilly Export’ are isolated for further analysis or consideration.
4.	Multiple anchor concepts are identified which are indirectly linked	The proposed algorithm is employed to identify the optimal path. Files associated with concepts retrieved in the optimal path are fetched, and the files common among all the concepts are given the most importance. In Figure 2 subfigure (c), let’s consider the search string ‘Fiber Export’ as an example. The search process identifies two Anchor Concepts: Fiber’ and ‘Agriculture Export’. These Anchor Concepts are connected by multiple paths, giving rise to various potential routes between them In this scenario, path Fiber, Jute, Jute Export, and ‘Agriculture Export are chosen and these concepts are further used by the retrieval process to fetch documents.

Table 2. TABLE II: Description of data sources

Data	Description	Type	Files	Size (0.8GB)
DS1	Districtwise fertilizer and crop production data, Karnataka.	Struct -ured	SQL	6%
DS2	Crop yield and seasonal crop production for each taluk, village and district in Karnataka	Struct -ured	CSV	59%
DS3	Agriculture domain data for taluks and districts in Karnataka.	Struct -ured	XLS, XLSX	11%
DS4	The files cover seasonal crop production for the years 2016 and 2017 in Karnataka.	Unstru -ctured	PDF	6%
DS5	The files explain about Data Project Specifications of land Utilisation Analysis	Unstru -ctured	DOC	1%
DS6	The image dataset covers information about millets, seasonal crops, and year-wise production in Karnataka	Unstru -ctured	JPEG, PNG	17%

Table 3. TABLE III: Description of Search Query

Query set	Description	Type of queries	Size	Users
QS1	The queries in this set are mainly descriptive. The query words are majorly part of the search space.	Direct search words	25	Domain Users
QS2	The queries are tagged with geo entities like Taluk, Village, or District in Karnataka.	Direct with Geo-tags	11	Domain Users
QS3	The queries are combinations of description, geo-tagging, and time parameters as year mentioned in yyyy format.	Direct with time tags	24	Domain Users
QS4	The description used in this set of queries may not be directly present in search space but semantically highly relevant to the domain.	Indirect query	6	Domain Users
QS5	These sets of queries are generated from use-cases from domain experts.	Direct-Indirect query	104	Domain Experts

Table 4. TABLE IV: Comparison of Type-2 Error

System	SemDR	Elastic Search	Lucene	Doc2Vec
Type-2 Error	11.9	88.1	80.2	96

Table 5. TABLE V: Analysis of ranked retrieval over baseline systems

	SDR	Elastic Search	Lucene	Doc2Vec
Top-3	75	75	80	30
Top-5	82	70	71	24
Top-7	86	61	65	20
Top-10	89	51	61	16

Equations

$$concept\_domain(c)=C{D}_c=Semantic\_Index[c]$$

$$J(C{D}_{C_1},C{D}_{C_2})=\frac{|C{D}_{C_1}\cap C{D}_{C_2}|}{|C{D}_{C_1}\cup C{D}_{C_2}|}$$

$$relation\_score(C_1,C_2)=1-J(C{D}_{C_1},C{D}_{C_2})$$

$$\sum _{\text{search queries}}\frac{(\text{Reference solution - retrieved documents})}{\text{reference solution}}$$

$$\sum _{\begin{array}{c}\text{search queries}\end{array}}\left|\frac{\text{Reference solution}-\text{retrieved documents}}{\text{top retrieved documents}}\right|$$

Full text

I Introduction

The Document Retrieval (DR) involves searching for and obtaining specific documents from a collection of heterogeneous data sources using a search term or query [1]. The selection of a retrieval mechanism is influenced by various factors, including the nature of documents, user needs, and available resources. Traditional approaches rely on techniques such as natural language processing, machine learning, statistical models, neural networks, and semantic-based models to retrieve relevant information. Keyword-based models rely on accurate matches between a user’s query and specific keywords or phrases in a document [2, 3, 4, 5, 6], while concept-based models [7, 8, 9] and semantic models [10, 11, 12, 13] aim to understand the underlying concepts behind a user’s query to give more accurate and comprehensive search results. Models used for retrieval include the Boolean model, Vector Space Model (VSM), probabilistic models, and language models pre-trained on extensive text data [14, 15, 16, 17, 18, 19]. Different models are often leveraged in combination to improve the accuracy and relevance of search results (In Section VI, a comprehensive analysis of the existing literature is presented). The motivation for this work stems from the challenges and limitations observed in traditional document retrieval approaches.

I-A Motivation

Let’s consider a scenario where a user wants to retrieve documents related to their search query ‘Fiber Export’. In this case, it is understood that the documents in the agriculture field are relevant to the search. While documents containing the words ‘Fiber’ or ‘Export’ would be an obvious choice, the ideal retrieval would focus on any type of fiber material being exported. The user’s main interest lies in obtaining files that capture export details of any fiber material.

To further illustrate this example, we can refer to the graph depicted in Figure 1. This graph represents important entities –in Section II, we define them as ‘concepts’– and their relationships within the agriculture domain. The graphical structure serves as a representation of domain-specific knowledge, showcasing the significance of semantics. The relevant concepts in the graph are denoted by dotted lines, indicating their membership in different groups. Identifying the concepts associated with ‘Fiber’ and ‘Export’ in the graph becomes more intuitive by considering the concepts belonging to Group ‘Thread Material’ and Group ‘Agriculture Trading’. Several paths link concepts from these two groups, and among them, Path 3 stands out as it directly connects Group ‘Thread Material’ and Group ‘Agriculture Trading,’ indicating concepts with a high semantic correlation to the search words. However, it is worth noting that Path 1 and Path 2, while connecting concepts from different groups (‘Types of crops’ and ‘Agricultural Trading’), include concepts that are not directly relevant to the search words. This example highlights the value of leveraging domain knowledge for semantic grouping and graph traversal, resulting in more effective retrieval of semantically aware results.

In the proposed research, we introduce a new method that utilizes a Concept Graph to represent domain knowledge[20]. This graph captures entities, and their interactions, presenting a comprehensive view of the domain. By leveraging the relationships and connections within the graph, our retrieval system can accurately identify and retrieve contextually relevant information that aligns with the user’s intent.

I-B Research Contribution

The key characteristics of our work are:

• •

The primary contribution of the research work is the novel algorithm- ‘Semantic-based Concept Retrieval using Group Steiner Tree’ that uses concepts and semantics.

• •

The proposed algorithm has a generic nature and can be extended to various applications, including entity resolution, information retrieval, text summarization, question-answering systems, compensation systems, and ontology alignment. In this paper, we specifically focus on expanding its application to document retrieval.

• •

The SemDR (Semantic-based Document Retrieval) system preserves heterogeneous data sources in their original forms while enabling context-relevant search. The proposed system is implemented and evaluated on real-world data111Data is received from the Government of Karnataka. The search queries are based on the needs of domain users and the result of these search queries are evaluated by domain experts and various users.

I-C Organization of the work

Continuing with the research narrative presented in Section I, Section II discusses the construction of a semantic concept graph that builds the foundation for the proposed algorithm discussed in Section III. Section IV delves into the implementation of the proposed algorithm, presenting both a logical design of the proposed SEMDR system and a case study that supports its implementation. Additional details on testing, benchmarking, and the findings of the study are provided in Section V. Section VI highlights a comprehensive analysis of related literature. Finally, the concluding segment of the paper summarizes the significant contributions made by the research and outlines potential pathways for future research and exploration.

II Semantic Concept Graph

To enhance document retrieval through semantic understanding, it is vital to grasp both the contextual aspects of the domain and the interconnections and arrangement of knowledge. Employing a graph-based structure to capture domain knowledge enables the organization of information in a structured manner and facilitates navigation through traversal. Our proposed method employs a graph-based approach for the document retrieval problem. We have added a semantic component to the graph, which establishes connections between concepts, resulting in the creation of a semantic concept graph that includes a semantic layer to identify concepts relevant for the retrieval. The captured semantics aid in retrieving pertinent documents.

II-A Building Semantic Concept Graph

One effective way to represent the information is by building a knowledge graph, comprising nodes representing entities and edges representing relationships [21], that provides a comprehensive representation of real-world entities and their attributes. Figure 2 sub-figure (a) illustrates the agriculture domain concepts, while sub-figure (b) provides a graphical representation of these concepts. The proposed document retrieval (DR) approach primarily focuses on identifying relevant concepts (both explicit and latent) within the domain context and their interconnections.

The explicit concepts derived from domain information are represented as nodes, and the hierarchical relationships (subclass or superclass) are known as contextual relations. We have considered a semantic proximity metric (defined later), which decides the degree of similarity or relatedness between concepts based on their meanings or semantic representations, to further reinforce the concept graph. Using this measure, concepts with similar properties are linked together, forming semantic relations. Additionally, concepts sharing common properties are grouped into latent concepts based on their commonalities; represented in red bold font in the Figure 2 sub-figure (c).

Semantic Concept Graph. A semantic concept graph is an edge-labelled graph $G=(C,R,W)$ where $C$ corresponds to a finite set of concepts, $R$ denotes a finite set of undirected edges between two concepts, and $W$ indicates weights associated with edges. The set of concepts $C$ is a collection of direct concepts and latent concepts. That is, $C=C_{direct}\cup C_{latent}$, where $C_{direct}$ is a concept derived from domain knowledge and $C_{latent}$ indicates a logical node that refers to a group of similar concepts. The set of relations $R$ is a collection of* contextual relations* and semantic relations. That is, $R=R_{contextual}\cup R_{semantic}$, where Contextual relation $R_{contextual}$ captures the hierarchical connections and dependencies among various concepts in the domain and semantic relation $R_{semantic}$ links the concepts with high semantic proximity. In this work, we determined the semantic proximity of two concepts using the Wu-Palmer Similarity score [22]. The semantic score varies between 0 to 1, where value 1 denotes the maximum similarity. The default threshold is set as 0.9 to achieve refined results.

Figure 2 (c) depicts the agriculture domain knowledge, including concepts such as cotton, silk, crops, and kharif. The blue-highlighted concepts are sub-concepts of the concepts denoted by bold black letters. In the figure, concepts are semantically grouped to define latent concepts. Each group represents a latent concept. For example, Group 1 represents the latent concept “agricultural transport”, encompassing concepts related to transporting agricultural goods. Group 2 connects various thread materials, including both plant-based and animal-based threads like silk and wool. Furthermore, Group 6, a subgroup within Group 2, categorizes specific concepts related to animal-based thread materials. The graph also incorporates additional latent concepts, which capture meaningful associations and relationships between concepts, enriching the semantic meaning and understanding of the graph.

II-B Assigning Weights to Semantic Concept Graph

After the construction of the semantic graph, we proceed to determine the weights assigned to each node and edge within the graph. These weight calculations can vary depending on the specific application. In our case, which involves document retrieval, we utilize two weight calculation functions, namely concept_domain and relation_score that are based on the content of the documents mapped to the concepts. Both these functions are defined later in Section IV. By leveraging the weights assigned to the graph’s edges, our DR algorithm will navigate the graph and retrieve relevant concepts based on their semantic similarities and relationships.

III Proposed Algorithm: Semantic-based Concept Retrieval using Group Steiner Tree

To identify the documents relevant to the search word, a crucial intermediate step is to determine the concepts that are most relevant to the search word. We propose Algorithm 1 to identify the concepts that closely align with the search word from the weighted semantic concept graph. The algorithm operates through the following steps:

• •

Identifying the anchor concepts: Determine the “anchor concepts” in the graph by identifying concept nodes that exhibit a semantic proximity score greater than 0.9222In this particular implementation set up, we established a threshold of 0.9 after experimenting with different values and taking into account the insights of domain experts. with the words in the search string.

• •

Identifying the latent concepts: By considering the Concept Graph containing Semantic Groups and Anchor Concepts.

Note: The task involves relaxing the Anchor concepts to encompass the respective group of concepts (latent concept) having anchor concept.

• •

Finding the relevant concepts. The key idea for identifying relevant concepts is that these nodes should be tightly connected to many anchor concepts. To formalize this intuition, we consider three factors: (i) the relevant documents lie on the intersection or union of anchor concepts, (ii) short paths are preferred, and (iii) paths with higher weights are better. These criteria are captured by the notion of a Steiner Tree.

Given an undirected and weighted concept graph $(C,R,W)$ with nodes $C$, edges $R$, and weights $\{w_{ij}\in W|w_{ij}\geq 0\}$ for the edge $(i,j)$ between nodes $i$ and $j$ and given a set $T\subseteq C$ of nodes called terminal nodes or anchor concepts, compute a tree $(C^{*},R^{*},W^{*})$ where $R^{*}\subseteq R,C^{*}\subseteq C$ that connects all terminals and has minimum cost in terms of total edge $*$ weights, i.e., $Min(\sum_{(i,j)\in R^{*}}w_{ij})$

In the case of two terminals, the solution can be obtained by finding the shortest path that connects two or more terminal nodes. However, our use case involves multiple terminals referred to as anchors. Additionally, these terminals are grouped into sets, where each set corresponds to a search keyword. In this scenario, it is sufficient to include at least one terminal from each set in the Steiner Tree. This problem, which encompasses these additional complexities, is commonly referred to as computing a Group Steiner Tree (GST) [23].

Given an undirected and weighted concept graph $(C,R,W)$ and given groups of terminal nodes $T$, for each $A\in T$ where $A\subseteq R$, compute the minimum-cost tree $(C^{*},R^{*})$ that connects at least one node from each $A\in T$ with minimum weighted path.

Technically, the objective is to discover a Group Steiner Tree (GST) that encompasses latent concepts, linking a maximal count of Anchor concepts while minimizing the associated expenses. The pertinent concepts that aid in connecting the anchor concepts are the concepts of interest because they are semantically linking the anchor documents which aid in fetching the relevant documents. The use cases discussed in Table I elaborate on the diverse considerations employed by the algorithm during the GST identification process.

III-A Proposed Algorithm

Since concept graphs can easily grow vast and complex, volume is one of the main issues to be addressed. As the amount of data grows, the performance may suffer, creating an unsatisfactory user experience and long response times. Traversing the entire graph for search interest would be unfeasible. To make the graph efficiently queryable, it is necessary to use traversal algorithms to facilitate seamless graph processing.

The suggested method centers around easing traversal by taking concept groups (latent concepts) into consideration. Initially, the algorithm identifies anchor concepts by assessing their semantic proximity, requiring a value greater than 0.9. The algorithm’s objective is to determine the potential concepts that establish links between these anchor concepts. The intuition is that these potential concepts, referred to as relevant concepts, could serve as common semantic points, sharing similar semantics among the interconnected anchor concepts. Consequently, these relevant concepts would be an ideal choice for identifying the relevant documents.

In order to identify the relevant concepts, an effective approach could involve combining the minimum spanning tree and the shortest path algorithms to establish an optimal path connecting the anchor concepts. The Steiner Tree problem, which is an optimization problem, is utilized to find the tree with the lowest cost that spans a specific set of terminal nodes (the anchor concepts). This tree includes all the terminal nodes and the edges that connect each terminal node to at least one other terminal node, and it may also incorporate additional nodes that contribute to connecting the terminal nodes. It is important to note that these additional nodes, which connect the terminal nodes, represent the potentially relevant concepts that we are seeking to identify.

Solving the Steiner Tree problem involves the algorithm exploring all conceivable combinations of edges and nodes to determine the optimal tree. However, as the problem size grows, the computational complexity becomes impractical due to the exponentially expanding solution space. Consequently, the Steiner Tree problem is categorized as NP-hard, implying the absence of any known polynomial-time algorithm capable of solving it exactly for all instances. As a result, achieving an optimal solution usually necessitates utilizing approximation algorithms or heuristics that yield solutions of good quality, although not guaranteed to be optimal, within a reasonable timeframe.

In this work we propose a heuristic-based method, with the Greedy algorithm. The Group Steiner Tree (GST) problem extends the traditional Steiner Tree problem by aiming to find a tree with the lowest cost that spans a group of concepts, referred to as groups or latent concepts. This generalization allows for the consideration of common connections among different groups, thereby minimizing the overall cost of the network. We define the Group Steiner Tree in context as follows:

Given a weighted semantic graph $G=(C,R,W)$, a set of terminal nodes $T$ ($T\subset C$) where a collection of latent concepts (or semantic groups) $L$ are defined as:

$T=\{c|c\in C\cap(\exists word\in Tokenized(Search\_String)\cap$ $semantic\_score(word,c)>0.9)\}$.

$L=\{S|S\subset C\cap(\exists d\in S|\forall c\in S\char 92\relax\{d\},semantic\_score(c,\\ d)>0.9)\}$

The goal is to identify a GST that spans across Semantic Groups, connecting a maximum number of Anchor concepts while minimizing the associated cost. The Anchor concepts, along with the additional concepts that connect them, are considered the Relevant Concepts of interest.

In the proposed algorithm 2, each Anchor Concept is extended starting to obtain the semantic group. Trees are developed iteratively by linking terminal nodes with the least-cost edges from each group. When common vertices are detected, trees are merged continuously. When a tree covers all Terminal nodes, the process is terminated.

The optimum tree may be a structure with Terminal nodes only. In such a case, the proposed algorithm will not generate a tree from semantic groups based on the GST. The number and distribution of terminal nodes in each group influence the complexity of finding a solution to the Group Steiner Tree problem[24].

III-B Complexity Analysis

The most straightforward brute-force approach to solving the Steiner Tree problem is to enumerate all possible subsets of vertices to find the minimum-cost tree. This approach has an exponential time complexity of $O(2^{n}\cdot n^{2})$, where $n$ is the number of vertices in the graph [25, 26]. Enumerating all subsets of vertices has a complexity of $O(2^{n})$, and for each subset, finding a minimum spanning tree has a complexity of $O(n^{2})$.

Let’s consider, the Group Steiner Tree algorithm operates on a graph with $n$ vertices and $m$ edges, along with a group $K$ containing $k$ vertices. To solve the problem, the algorithm calculates Steiner trees for each vertex in the group, connecting them to the rest of the vertices in the graph. The time complexity of this step relies on the chosen algorithm, typically around $O(2^{k}\cdot n^{2})$. The algorithm then merges these Steiner trees with a main tree, taking $O(n^{2})$ time. Processed vertices are subsequently removed from the group, which can be done in $O(k)$ time. These steps are repeated until the group is empty, with the loop having an overall time complexity of $O(k\cdot(2^{k}\cdot n^{2}+n^{2}+k))$. Finally, the minimum group Steiner tree is returned in constant time, resulting in an overall time complexity of approximately $O(k\cdot 2^{k}\cdot n^{2})$. It’s worth noting that these complexities are estimates and may vary based on specific algorithms and heuristics employed.

IV SemDR: Semantic-based Document Retrieval

SemDR refers to a semantic-based document retrieval system that leverages the proposed algorithm to retrieve documents based on their semantic relationships. By incorporating domain information and employing semantic grouping through the Group Steiner Tree approach, SemDR aims to identify relevant concepts and improve the accuracy and comprehensiveness of document retrieval. It utilizes a knowledge graph constructed from domain-specific information and real-world data to enhance the retrieval process. SemDR is designed to overcome the limitations of traditional text-based search methods and provide more precise and meaningful search results for users. SemDR is typically composed of an Index generation process, Document Access, and Query processing. The indexing system generates an index structure by evaluating each document’s text and finding key phrases and concepts. Documents are accessed via the index structure. The query processing system is responsible for the task of processing user queries over documents retrieved. To identify documents that best match the user’s query, the system often includes a collection of algorithms like keyword matching, and semantic analysis.

The proposed algorithm - ‘Semantic Concept Retrieval by GST’ discussed in section III, is applied in a SemDR system. The key component of SemDR is the discovery of semantically relevant concepts and their utilization in document retrieval. Figure 3 depicts an overview of the SemDR system. There are three modules in total:

1. The Semantic Pipeline is concerned with providing the framework for semantic data retrieval.

2. Semantic concept retrieval; this module is based on the algorithm we proposed. The module’s finding of relevant concepts aids in the retrieval of related documents.

3. The query module enables the user to run structured linked queries over the obtained documents.

IV-A Architecture overview

Semantic-based data pipeline module is responsible for generating index structure over heterogeneous data. The semantic index structure generates a semantic layer over data sources. All document retrieval requests are processed via index structure.

Data sources: Table II highlights the data sources contributing to the SemDR system. Two types of data sources are considered for implementation- structured and unstructured. The primary goal of the SemDR system is to pertain data sources in their original format while facilitating the retrieval mechanism. We extract the metadata about the document context to enable heterogeneous data for processing. The metadata captures the contextual characteristics of the document. In structured documents, it comprises attributes, domain, range, and description; unstructured documents primarily include frequent words and descriptions. The metadata acts as a logical representation of documents, which is used by the semantic mapping process to index the documents.

Domain Knowledge extraction: Semantic Concept Graph $G$ constitutes the foundation of our proposed algorithm and the SemDR system. The $G$ is built over the data gathered from domain knowledge. The subject matter expert compiles his years of expertise and insight into a document, report, or article. These artifacts aid in the development of domain knowledge. Government documents, survey information, performance reviews, and proposals are examples of artifacts that add to domain knowledge. $SPO$ triples representation is the most widely used representation to relate raw data, domain knowledge and semantics. $SPO$ triples are composed of three elements: The subject $S$ denotes the context. Predicate $P$ defines the relationship or attribute that characterizes the subject to the object $O$. $SPO$ triples are used to represent data in a machine-readable manner, giving systems the ability to deduce the relationships between entities. RDF/XML, Turtle, and JSON-LD are a few file types that can hold SPO triples. These file types can transfer data between systems or store data, making it simple for individuals to use. Additionally, $SPO$ triples can be stored in CSV (Comma Separated Values) file form. Each $SPO$ triple is displayed in this format as a row in the file, with commas separating the subject, predicate, and object values.

The SemDR system accepts CSV and OWL formats to leverage the domain knowledge. The domain knowledge includes information for each entity, such as words that describe the entity, linked entities, and relationships. SemDR captures domain knowledge in $SPO$ format using CSV or OWL formats. SemDR treats subjects and objects as concepts and the description, superclass and subclass relationship among concepts as predicates. The information captured is referred to as concept information is further used for semantic mapping purposes.

Semantic mapping: The semantic mapping stage is critical in constructing the semantic index structure. The index structure allows access to data while also contributing to creating a concept graph. The primary goal of semantic mapping is to connect documents to concepts. The semantic mapper considers two inputs, the first input is concept information and another input is document metadata. We can relate texts to concepts using the semantic similarity function. However, the goal here is not just to return all relevant documents but also to allow users to discover relevant concepts linked to their search query. The SemDR system proposes the clustering approach to group together documents that share common characteristics or themes, enabling us to identify new concepts and patterns within a collection of documents.

Clustering: In document retrieval, K-means clustering is used to group documents with similar content or features, streamlining the organization and retrieval of pertinent information. This approach facilitates the identification of document clusters sharing common themes or topics, enhancing the effectiveness of information retrieval processes.

If ingestion is considered to be a one-time process; grouping ingested documents and then selecting top words in each cluster to map them to concepts using a semantic similarity might be an effective option. However, given the dynamic nature of ingestion, the system should continue to accept new documents, which would result in changes to the clusters and index structure. A static generic cluster would be useful in overcoming this difficulty. We used Wikipedia data dump and basic k-means clustering to identify generic clusters in this implementation. We also found the top terms that define each cluster.

The key observation here is that any given corpus can be utilized to discover generic clusters. However, the specific approach to clustering, the decision on the number of clusters, and the selection of the top words representing a cluster may vary depending on the specific domain of interest.

The Figure 4 depicts the semantic mapping block in more detail. The semantic block is given domain knowledge containing concepts and their descriptions and metadata defining documents. The static clusters serve as the anchor element. Documents are mapped to clusters, and concepts are mapped to clusters, using a semantic similarity function that generates concept-to-document mapping as a semantic index structure.

IV-B Semantic Graph Construction:

This block is responsible for building the concept graph. Section II has elaborated on the intuition involved in the process of graph construction. The domain knowledge is used to create a basic concept graph in which each concept is represented as a node, and the predicate defines the edge. The graph generated is a bidirectional unweighted graph. In this implementation, concept graph includes 327 concepts, and almost 86,000 edges. To assign weights to the graph, the system uses a generated index structure. Using the concept_domain function, we compute a score for each concept. For each concept c in domain knowledge, the function $concept\_domain$ is defined as

[TABLE]

The function value is represented by the set of documents associated with each concept. We define the function relation_score to determine the score for every edge using inverse Jaccard similarity or the Jaccard index. It is a measure of how distinct the two groups of elements are. It represents the value of elements that differ between the two sets in relation to the total number of distinctive objects in both groups. The Jaccard similarity coefficient can be defined as,

[TABLE]

where $C_{1}$,$C_{2}$ are concepts and $CD_{c_{1}}\,CD_{C_{2}}$ holds the set of documents associated with respective concepts defined by $concept\_domain$ function. The primary goal of assigning weights to relationships is to select concepts with the highest number of documents linked with them while deciding on the Group Steiner Tree in algorithm 2 described in Section III. In this application of document retrieval, the major focus is on identifying the nodes having a maximum number of associated documents. To achieve this over shortest path finding algorithms, We use the Inverse Jaccard similarity coefficient to define the relation_score. The inverse score assures the selection of concepts with a greater edge weight. The Inverse Jaccard similarity coefficient is defined as 1 minus the Jaccard similarity coefficient. The function relation_score can be defined mathematically as follows:

[TABLE]

The Semantic Concept Retrieval block receives the weighted semantic concept graph and applies the proposed algorithm in order to obtain the relevant concepts. The relevant concepts are forwarded to the data access block to retrieve the documents using a semantic index structure. The semantic closeness of the retrieved documents to the search input is used to rank the documents.

IV-C Query Module:

Given that the data sources are heterogeneous, the retrieved and ranked documents may be in different file formats. In this study, we restrict ourselves to solely querying structured documents. The structured documents include CSV, XLS, XLSX, and SQL data files. These structured documents are semantically connected for querying. Based on feasibility, the documents that have been retrieved are merged. The merged documents are then considered for the join operation. The equivalent of a relational JOIN is performed between data files only if there is any link between them that is equivalent to a PRIMARY KEY and FOREIGN KEY relationship. The column name, attribute range, attribute domain values, and semantic similarity are all considered to determine the link condition. If attributes match the factors mentioned above, then the documents are considered for join queries. The result is shown in both its visual representation and tabular form.

V Results and Discussion

The objective of this section is to provide an overview of the conducted experiments, including details on the utilized datasets, and evaluation metrics. Furthermore, the significance of benchmarking is emphasized, as it enables an objective comparison with baseline systems. This section is organized around the following key points: A. Experimental Setup, B.Baseline Systems, C.Evaluation Metrics, D.Result and Observations.

V-A Experimental Setup

We compare the results of the proposed system with the outcomes of the baseline systems. We are considering a keyword-based retrieval system and a semantic-based retrieval system as a baseline. Our experimental evaluation is limited to Lucene, ElasticSearch, and Doc2Vec tools. The reference solution is built considering the application of the proposed research. The domain experts have asked to select files from the search corpus for all the search strings. These results are considered reference solutions to compare with baseline systems and the proposed approach.

V-A1 Search Query

The search query represents the user’s search interest in retrieving documents. We have collected the search queries from domain experts to test the recommended system considering the combination of search interests including geographical location, time parameter and search strings. The search queries are gathered using Google Forms. To accommodate search strings based on standard test cases relevant to the domain, we refer to use-cases provided by government use-case analysis333NDAP-https://ndap.niti.gov.in/. Table III contains information about the search queries. The search strings can be a combination of domain interest, time period of interest, and geographical location (state/district/village).

To understand the performance of the proposed system in more detail, we group search queries as follows: keywords appearing in the dataset called Direct search words, keywords with geographic location, keywords with the time period, and strings that are not appearing in the dataset but semantically relevant to the dataset called as Indirect search words. All of these types contribute to a total of 170 search strings.

V-A2 Reference Solution

We consider public data for implementation, as indicated in Table II. The information is gathered from multiple government departments. Our search query corpus and concept graph consider domain experts and domain users as sources of input because they are the primary sources of authentic information. In SemDR, the reference solution is built over a query set as discussed in Table III. The domain experts selected the files of interest by considering metadata and content. The search query and the set of relevant files selected by the domain expert contribute to the reference solution.

V-B Baseline Systems

We opted for the following three baseline systems for the performance analysis of SemDR:

Lucene is an open-source search library written in Java that provides a collection of low-level APIs for indexing and searching data [27]. Lucene allows users to index documents by analysing the content and constructing a keyword-based search index [28].

Elasticsearch leverages Lucene as its fundamental search engine with additional capabilities for distributed search, real-time search, analytics, and queries in domain-specific languages [29].

Doc2Vec is a natural language processing technique that takes the Word2Vec algorithm and applies it to entire documents. Document embeddings are dense vector representations of whole documents in the same high-dimensional space that Doc2Vec generates. The vector space approach can also be used to find relevant materials [30].

V-C Evaluation Metric

To validate the results, we analyze conventional parameters such as precision, accuracy, recall, F1-score, and error score against each search string. To compute all of the above parameters, the manual selection results by the experts are used as actual values, while the baseline or SemDR system results are used as predicted values.

The confusion matrix is built on the calculation of the following values,

True Positive (TN) indicates the count of documents retrieved by the manual selection process and other baseline systems.

False Positive (FP) indicates the number of documents retrieved by the baseline system but not by the manual selection process.

False Negative (FN) indicates the number of documents retrieved by the manual selection process but not by the baseline system.

True Negative (TN) indicates the count of documents neither retrieved by the baseline system nor by the manual selection process.

Error Type. When a document retrieval system retrieves irrelevant documents that shouldn’t have been retrieved, the error is known as a type 1 error. In other words, it is a false positive error when a search result includes a document that is not related to the user’s search. The type 2 error occurs in document retrieval when the system fails to retrieve a relevant document that should have been retrieved. It is a false negative error in which a document that is related to the user’s query is not included in the search results.

The performance metrics derived from the confusion matrix are:

Accuracy: In document retrieval systems, accuracy measures the percentage of retrieved documents relevant to the query. More formally, accuracy is the ratio of correctly retrieved documents to the total number of retrieved documents. $Accuracy=(TP+TN)/(TP+TN+FP+FN)$

Precision: Precision measures the proportion of retrieved documents relevant to the query. It is the ratio of correctly retrieved documents to the total number of retrieved documents. In other words, precision measures how precise the system is in returning only relevant documents. $Precision=TP/(TP+FP)$

Recall: Recall measures the proportion of relevant documents retrieved by the system. It is the ratio of correctly retrieved documents to the total number of relevant documents. Recall measures how effective the system is in retrieving all relevant documents. $Recall/Sensitivity=TP/(TP+FN)$

F1 score: F1 score is a measure that combines precision and recall to provide a single metric that balances both measures. It is the harmonic mean of precision and recall. $F1-score=2*((precision*recall)/(precision+recall))$

We consider the above parameters to understand the overall performance of the system where accuracy measures the overall effectiveness of the system, precision measures the proportion of relevant documents returned, recall measures the proportion of relevant documents retrieved, and the F1 score combines precision and recall to provide a balanced measure of the system’s effectiveness.

Within the context of evaluation, we have undertaken the conversion of metrics such as precision, recall, accuracy, and F1-score into a percentage scale. This conversion serves to facilitate the interpretation and comparative analysis of performance for various experiments across baseline systems.

V-D Result and Observations

Figure 5 compares the baseline systems to the SemDR system. The trend line depicts a reference solution against 170 search keywords. The graph indicates that the red/orange dots denoting the results of the SemDR system follow the trend line, whilst other systems retrieve fewer documents being clustered near the X-axis.

V-D1 Evaluation of SemDR using conventional performance parameters

The graph in Figure 6 indicates the performance evaluation of four different systems (SemDR, Elastic Search, Lucene, and Doc2Vec) based on four different evaluation metrics (precision, recall, accuracy, and F1-score). To ensure a standardized comparison scale, all values are expressed as percentages. The performance of SemDR is quite good, with precision and recall at 90% and 88%, respectively, accuracy at 82%, and an F1-score of 89%. This indicates that SemDR retrieves a high percentage of relevant documents with a good balance of precision and recall. On the other hand, Elastic-Search, Lucene, and Doc2Vec have lower precision, recall, accuracy, and F1-score values. Elastic-Search has the highest precision among these three systems but with a low recall, indicating that it retrieves only a small number of relevant documents. Lucene and Doc2Vec have lower precision and recall values than Elastic-Search, and they also have lower accuracy and F1-score values, indicating poor overall performance.

V-D2 Error Calculation:

In general, it is ideal for minimizing both type 1 and type 2 errors. But some error types could be more severe than others, depending on the application. It is challenging to guarantee the accuracy of the knowledge used in implementing SemDR because it is based on domain knowledge provided by subject matter experts. Additionally, the actual outcomes used for evaluation are interpreted manually, which introduces drift into the results. Therefore, it is essential to comprehend the result with documents neither retrieved by the baseline nor by the proposed system. In Figure 5, the error is represented by the zoomed section. To understand the trend of not retrieved documents, type 2 error or false negative error is considered. With reference to Table IV, the error value measures how often the system produces incorrect results, expressed as a percentage of the total number of queries processed. The mean of Type-2 (False Negative) Error is calculated as

[TABLE]

It is observed that Lucene has an error value of 80.2%, ElasticSearch has an error value of 88.1%, Doc2Vec has a value of 96%, and SemDR has a value of 11.9%. The SemDR system has the lowest error rate, which means it has the highest accuracy among the four systems.

V-D3 Comparison with baseline systems

Figure 6 illustrates the True Positive value over different baseline systems with respect to different search strings with location and time tags. It is observed that the overall performance of the systems varied across query types. SemDR had the highest overall performance, retrieving relevant results for the queries across all query types. Lucene and Elastic Search also performed well, particularly for Direct Query searches with additional location and time tags (QS2 and QS3). Doc2Vec had relatively low performance across all query types.

Figure 6 illustrates the performance parameters to comprehend how different systems perform with respect to different search strings with location and time tags. The orange line indicates the trend of the SemDR system. All the graphs capture various parameters over a query set QS1, QS2, QS3, QS4 and QS5. It is observed that the orange line trends achieve better values over all the parameters. Referring to the graph in Figure 6, we can see that SemDR performs the best overall, with the highest scores in precision, recall, accuracy, and F1-score across all five types of queries.

V-D4 Evaluation with ranked retrieval

Document retrieval is an ideal application for managing systems with massive amounts of data. The ranking mechanism would help users focus on important documents based on search interest. The retrieved documents are organized using ranking algorithms based on their relevance to the user’s input, allowing the user to query the most pertinent data rapidly. To define the ranking, the proposed method uses the cumulative score of the semantic similarity function across the search string and document metadata. All the considered baseline systems generate ranked retrieval of the documents. As the reference system is based on manual intervention, the rank of the document is a subjective value and would vary for each domain expert. Hence, it is critical to calculate the overall ranking error.

In our experimental evaluation, we try to choose top-k ranked documents and check if they exist in the result set of the reference system. This validates the baseline systems over ranked retrieval. We analyzed the top 3, 5, 7, and 10 retrieved documents for comparison. To evaluate the ranking of documents over 170 queries, the mean of True Positive values in percentage is calculated as,

[TABLE]

Table V indicates that as the number of considered documents increases from 3 to 10, the percentage of relevant documents retrieved by other baseline systems decreases except the SemDR system signifying the proposed system retrieves more relevant documents.

V-D5 Additional ontological Constructs

To adapt the proposed algorithm for document retrieval over government data, we observe that search strings include various geographic locations and time references to refine the search interest. It is also observed that the search queries mention Village, District, or Taluk to refine their search interest. The logical representation of the geography of India includes a hierarchy of (Country-State-District-Taluk-Village)444The generic geographic hierarchy is Country-State-County-Village, here Taluk can be considered equivalent to County. We also capture information about neighboring Villages, Talukas, and Districts. To support geographically tagged search, we captured geographic information in the form of the ontology using shape files and constructed separate index structures. We preferred to keep the domain ontology and geographic ontology separate for the following reasons,

• •

The considered Agricultural domain knowledge is generic. Still, the data used for ingestion and search queries are geographically tagged and relevant to Karnataka.

• •

The generic domain ontology prefers identifying semantically grouped concepts, but the geographic ontology holds facts and hierarchical structure.

• •

To identify the alternate concepts in the geographic ontology, we prefer traversing to the neighboring node or the parent node and not a semantically relevant node.

VI Related Literature

The choice of a specific retrieval mechanism is influenced by multiple factors, including document characteristics, user requirements, and available resources for system development. Depending on the search corpus, cross-lingual, text-based, or image-based systems can be employed. Traditional methods utilize diverse techniques like natural language processing, machine learning, statistical models, neural networks, and semantic-based models to extract pertinent information.

Approaches for Retrieval: Traditional keyword-based search relies heavily on exact keyword matches [2]. Conversely, Google’s BERT model for NLP considers contextual meaning, enhancing keyword-based search [3, 4]. Research [5] introduces a BERT-based model with adaptive weighting for query terms, outperforming traditional methods treating all terms equally. For efficient keyword-based retrieval on large-scale text data [6], a two-stage approach is proposed, initially retrieving candidates with an inverted index and then ranking using machine learning. However, keyword-based models are prone to synonym and homonym challenges, potentially missing crucial data. Concept-based models aim to understand query concepts beyond textual context, offering more accurate and comprehensive results. A novel neural network-based approach [7] combines traditional vector space models with neural embeddings. A hierarchical network-based approach [8] captures global and local document information. Another hybrid approach [9] fuses semantic knowledge graphs with vector space models. Nonetheless, concept-based models may lack coverage for general concepts as they’re often trained on specific datasets. Semantic models capture rich semantic connections, including synonyms and antonyms, enabling thorough and precise results, especially for complex searches [10, 11]. Research [31] proposes a semantic-based retrieval method using WordNet and ontology. Another work [12] employs knowledge graphs for entity relationships. The Multi-Level Semantic Matching approach [13] combines various features for similarity computation, often employing multiple models for improved accuracy [32, 33, 34, 35].

**Retrieval Models: **Various models are employed in document retrieval, including: Boolean Model: It uses Boolean logic to match documents with queries but lacks ranking and term weighting, impacting retrieval effectiveness [14, 15, 36, 37]. Vector Space Model (VSM) assigns terms unique dimensions and computes document-query similarity based on vector angles [16, 17]. However, it struggles with semantic connections between terms [38]. Probabilistic Models employ probability theory to estimate document relevance based on term frequencies and document characteristics [39, 19]. NLP and Language Models are pre-trained language models that learn contextualized representations for queries and documents, enhancing retrieval [40, 41, 42]. Neural models extend these capabilities [43, 44, 45]. Machine Learning Models utilize labelled data to learn patterns and optimize parameters for retrieval [18, 46]. Knowledge Graph (KG) Model represents information as a graph with entities as nodes and relationships as edges, offering structured knowledge for retrieval [47, 48].

KG for Retrieval: Knowledge graphs enhance semantic document retrieval by capturing meaning, providing context, supporting machine learning models, and offering advanced search capabilities [49, 50, 51, 52, 23, 53, 54].

**Existing Search Engines: **While Document Retrieval (DR) and Information Retrieval (IR) are distinct, ”Search Engines” represent another related branch widely used in web applications [55]. Semantic search engines, like Semantic Search-Nordlys, leverage various technologies for entity recognition, knowledge base linking, and relevant entity retrieval [56]. The SCORE platform creates a large knowledge base using semantics [57]. Schema.org facilitates ontology-driven Semantic Search [58]. Existing search engines, however, may struggle with complex queries and semantic nuances, impacting result accuracy.

The literature extensively covers retrieval approaches and models. However, the absence of domain-specific context in retrieval can lead to misleading outcomes.

VII Conclusion

The research work introduces SemDR, an innovative semantic-based document retrieval system designed for heterogeneous data sources. The retrieval process is motivated by domain-specific information, which is harnessed through the process of concept graphs and the semantic linking of concepts. The system optimizes retrieval by employing semantic-driven concept identification using the Group Steiner Tree approach. This technique aids in pinpointing pertinent concepts and utilizing them for document retrieval.

The query module of SemDR facilitates the integration and querying of structurally relevant documents with semantic coherence. Notably, the SemDR system exhibits remarkable performance compared to conventional keyword-based retrieval systems and existing semantic-based document retrieval systems, attaining an accuracy rate of 90% and a precision score of 82%. As part of future work, the aspiration is to evolve SemDR to adapt seamlessly across diverse domains.

Acknowledgments

This work was supported by Karnataka Innovation & Technology Society, Dept. of IT, BT and S&T, Govt. of Karnataka, India, vide GO No. ITD 76 ADM 2017, Bengaluru; Dated 28.02.2018. The research team is also grateful to the Government of Karnataka, the IIIT Bangalore Center for Open Data Research (CODR), Bengaluru, India, for their significant data and domain expertise collaboration.