Model Matching Challenge: Benchmarks for Ecore and BPMN Diagrams

TL;DR

This paper introduces benchmarks for model comparison tools in Model Driven Engineering, focusing on Ecore and BPMN diagrams, by analyzing failure cases and proposing evaluation standards.

Contribution

It presents five edit operations causing comparison failures and offers benchmarks to assess the quality of model matching algorithms.

Findings

Identification of five key edit operations affecting comparison quality

Analysis of failure reasons in current comparison algorithms

Proposed benchmarks for evaluating model matching tools

Abstract

In the last couple of years, Model Driven Engineering (MDE) gained a prominent role in the context of software engineering. In the MDE paradigm, models are considered first level artifacts which are iteratively developed by teams of programmers over a period of time. Because of this, dedicated tools for versioning and management of models are needed. A central functionality within this group of tools is model comparison and differencing. In two disjunct research projects, we identified a group of general matching problems where state-of-the-art comparison algorithms delivered low quality results. In this article, we will present five edit operations which are the cause for these low quality results. The reasons why the algorithms fail, as well as possible solutions, are also discussed. These examples can be used as benchmarks by model developers to assess the quality and applicability of a model comparison tool for a given model type.