Enriching Social Science Research via Survey Item Linking

TL;DR

This paper introduces Survey Item Linking (SIL), a two-stage task to automatically connect survey questions mentioned implicitly in social science texts to a knowledge base, improving research referencing and comparison.

Contribution

It defines the SIL task, creates a high-quality dataset for it, and benchmarks deep learning models, highlighting challenges and directions for future improvements.

Findings

SIL is feasible with deep learning models.

Errors mainly occur in mention detection, especially with multi-sentence context.

End-to-end modeling and more diverse data can improve performance.

Abstract

Questions within surveys, called survey items, are used in the social sciences to study latent concepts, such as the factors influencing life satisfaction. Instead of using explicit citations, researchers paraphrase the content of the survey items they use in-text. However, this makes it challenging to find survey items of interest when comparing related work. Automatically parsing and linking these implicit mentions to survey items in a knowledge base can provide more fine-grained references. We model this task, called Survey Item Linking (SIL), in two stages: mention detection and entity disambiguation. Due to an imprecise definition of the task, existing datasets used for evaluating the performance for SIL are too small and of low-quality. We argue that latent concepts and survey item mentions should be differentiated. To this end, we create a high-quality and richly annotated dataset consisting of 20,454 English and German sentences. By benchmarking deep learning systems for each of the two stages independently and sequentially, we demonstrate that the task is feasible, but observe that errors propagate from the first stage, leading to a lower overall task performance. Moreover, mentions that require the context of multiple sentences are more challenging to identify for models in the first stage. Modeling the entire context of a document and combining the two stages into an end-to-end system could mitigate these problems in future work, and errors could additionally be reduced by collecting more diverse data and by improving the quality of the knowledge base. The data and code are available at https://github.com/e-tornike/SIL .