Replication and Generalization of PRECISE

TL;DR

This paper replicates and formalizes the PRECISE system, revealing that initial results did not fully replicate but providing a clearer framework for future improvements and extensions.

Contribution

It offers a formal description and implementation of PRECISE, clarifies its approach, and suggests a roadmap for enhancing the system.

Findings

Initial replication did not fully reproduce PRECISE results

Formalization improved understanding and implementation

System performance shows significant improvement after formalization

Abstract

This report describes an initial replication study of the PRECISE system and develops a clearer, more formal description of the approach. Based on our evaluation, we conclude that the PRECISE results do not fully replicate. However the formalization developed here suggests a road map to further enhance and extend the approach pioneered by PRECISE. After a long, productive discussion with Ana-Maria Popescu (one of the authors of PRECISE) we got more clarity on the PRECISE approach and how the lexicon was authored for the GEO evaluation. Based on this we built a more direct implementation over a repaired formalism. Although our new evaluation is not yet complete, it is clear that the system is performing much better now. We will continue developing our ideas and implementation and generate a future report/publication that more accurately evaluates PRECISE like approaches.