Effect of Tuned Parameters on a LSA MCQ Answering Model

TL;DR

This study investigates how tuning parameters of Latent Semantic Analysis affect its ability to answer French biology MCQs, revealing that optimized low-dimensional semantic spaces can perform comparably to students.

Contribution

It introduces a method for fine-tuning LSA parameters and a novel entropy-based weighting scheme to improve MCQ answering performance.

Findings

Optimized LSA parameters yield performance comparable to 7th and 8th grade students.

Low-dimensional semantic spaces can be effective despite limited training data.

Entropy-based weighting of answer terms is crucial for model success.

Abstract

This paper presents the current state of a work in progress, whose objective is to better understand the effects of factors that significantly influence the performance of Latent Semantic Analysis (LSA). A difficult task, which consists in answering (French) biology Multiple Choice Questions, is used to test the semantic properties of the truncated singular space and to study the relative influence of main parameters. A dedicated software has been designed to fine tune the LSA semantic space for the Multiple Choice Questions task. With optimal parameters, the performances of our simple model are quite surprisingly equal or superior to those of 7th and 8th grades students. This indicates that semantic spaces were quite good despite their low dimensions and the small sizes of training data sets. Besides, we present an original entropy global weighting of answers' terms of each question of the Multiple Choice Questions which was necessary to achieve the model's success.