# SEMA: an Extended Semantic Evaluation Metric for AMR

## TL;DR

This paper introduces SEMA, an improved semantic evaluation metric for AMR that addresses limitations of the existing smatch metric, providing a more refined, robust, fairer, and faster evaluation method.

## Contribution

The paper presents SEMA, an extended AMR evaluation metric that overcomes smatch's drawbacks, enhancing accuracy and fairness in parser assessment.

## Key findings

- SEMA outperforms smatch in robustness and fairness.
- SEMA is faster than smatch.
- Evaluation with four AMR parsers demonstrates SEMA's effectiveness.

## Abstract

Abstract Meaning Representation (AMR) is a recently designed semantic representation language intended to capture the meaning of a sentence, which may be represented as a single-rooted directed acyclic graph with labeled nodes and edges. The automatic evaluation of this structure plays an important role in the development of better systems, as well as for semantic annotation. Despite there is one available metric, smatch, it has some drawbacks. For instance, smatch creates a self-relation on the root of the graph, has weights for different error types, and does not take into account the dependence of the elements in the AMR structure. With these drawbacks, smatch masks several problems of the AMR parsers and distorts the evaluation of the AMRs. In view of this, in this paper, we introduce an extended metric to evaluate AMR parsers, which deals with the drawbacks of the smatch metric. Finally, we compare both metrics, using four well-known AMR parsers, and we argue that our metric is more refined, robust, fairer, and faster than smatch.