A Pragmatic Approach for Measuring Maintainability of DPRA Models

TL;DR

This paper introduces a quantitative approach for assessing the maintainability of DPRA models created with PyCATSHOO, using metrics from software engineering to improve early decision-making in model development.

Contribution

It proposes a novel set of metrics for evaluating DPRA model maintainability and demonstrates their effectiveness with a practical case study.

Findings

Metrics can serve as early indicators of model modifiability.

Selected metrics help assess complexity and maintainability.

Approach supports better decision-making during model development.

Abstract

Dynamic Probabilistic Risk Assessment (DPRA) is a powerful concept that is used to evaluate design and safety of complex industrial systems. A DPRA model uses a conceptual system representation as a formal basis for simulation and analysis. In this paper we consider an adaptive maintenance of DPRA models that consist in modifying and extending a simplified model to a real-size DPRA model. We propose an approach for quantitative maintainability assessment of DPRA models created with an industrial modeling tool called PyCATSHOO. We review and adopt some metrics from conceptual modeling, software engineering and OO design for assessing maintainability of PyCATSHOO models. On the example of well-known "Heated Room" test case, we illustrate how the selected metrics can serve as early indicators of model modifiability and complexity. These indicators would allow experts to make better decisions early in the DPRA model development life cycle.

Tables

Table 1. Table 1 : PyCATSHOO models of the Heated Room test case

Model description:	Set 1 (Design 1)	Set 2 (Design 2)
Initial model: 1 heater + 1 room	Case 0	Case 0a
System level modificaiton: 4 independent heaters H0..H3 + 1 room	Case 1	Case 1a
Component level modification: standby redundancy of heaters: H0 - main; H1..H3 - backups	Case 2	Case 2a

Table 2. Table 2 : Metrics for Maintainabiity Assessment of PyCATSHOO models

Metrics:	Tool support:
LOC (lines of code), RLOC (relative modifications)	cloc, radon
Cyclomatic Complexity (CC)	radon
Halstead (vocabulary, volume, effort, bugs, difficulty)	radon
Maintainability Index (MI)	radon

Table 3. Table 3 : RLOC comparison: Original vs. Alternative design

Set 1	RLOC	Set 2	RLOC
$Case0⟶Case1$	82,25%	$Case0a⟶Case1a$	18,57%
$Case1⟶Case2$	108,69%	$Case1a⟶Case2a$	83,4%

Equations

$$RLO{C}_{case}=\frac{LO{C}_{modif}+LO{C}_{add}+LO{C}_{rem}}{LO{C}_{case}}$$

$$N=N_1+N_2$$

$$\eta ={\eta }_1+{\eta }_2$$

$$V=N\times {\log }_2\eta$$

$$D=\frac{{\eta }_1}{2}\times \frac{N_2}{{\eta }_2}$$

$$E=D\times V$$

$$B=\frac{V}{3000}$$

$$MI=171-5.2\times \ln (V)-0.23\times CC-16.2\times \ln (LOC)$$

Full text

1 Introduction

Dynamic Probabilistic Risk Assessment is a powerful concept that is used to evaluate design and safety of complex systems where the static reliability methods like fault trees find their limits [2]. A DPRA model combines the characteristics of conceptual model and a software application: it formally describes some aspects of the physical world (for example, a complex mechanical system) for purposes of understanding and communication [28]; it serves a formal basis for further system simulation and analysis. For feasibility, proof of concept, algorithm benchmarking and other preliminary studies simplified DPRA models are used. Adaptive maintenance is an important part of a DPRA model life cycle: it consists in modifying and extending a simplified model to a real-size DPRA model.

In this work, we propose an approach for maintainability assessment for DPRA models created with an industrial modeling tool called PyCATSHOO. We define a set of metrics that can serve as early indicators of PyCATSHOO model modifiability and complexity.

PyCATSHOO is a tool dedicated for dependability analysis of hybrid systems developed and used by EDF [8, 9]. PyCATSHOO models are executable modules that can be written in Python or C++ and interpreted by PyCATSHOO engine.

We review some well-known metrics from conceptual modeling, software engineering and object-oriented design, including size measures (LOC), complexity measures (McCabe’s Cyclomatic Complexity [24]), lexical measures (Halstead [17]), Maintainability Index (MI) [30] and OO-specific measures [7]. Based on our review, we select and adapt a set of metrics applicable to DPRA models and PyCATSHOO models in particular.

We make an assumption that the selected metrics can show a difference between two PyCATSHOO model designs already at the early model development life cycle, helping experts to make better decisions.

To validate this assumption, we apply selected metrics to assess and compare two designs of the Heated Room system. Heated Room is a well-known test case reported in [4, 5]. It describes a system that consists of a room and a heater that can switch on and off to maintain the ambient temperature. This example illustrates a hybrid system that combines deterministic continuous phenomena (i.e., temperature evolution) and stochastic discrete behaviour (i.e., functioning of a heater).

We create two sets of PyCATSHOO models as illustrated in Table1: Set 1 follows the original design ideas from [8], Set 2 represents an alternative model design promoting the low coupling design principle. The corresponding models in two sets are semantically equivalent, i.e., they demonstrate the same simulation traces. We compare measurements for the model sets and discuss the results.

The reminder of this paper is organised as follows: Section 2 presents DPRA models and PyCATSHOO modeling tool; Section 3 discusses the related works on maintainability assessment; Section 4 presents our approach for maintainability assessment of PyCATSHOO models; In Section 5 we describe two alternative designs of the PyCATSHOO models for the Heated Room test case. We assess maintainability of these designs and discuss our results in Section 6; Section 7 presents our conclusions.

2 DPRA Models and PyCATSHOO Modeling Tool

EDF has a long-standing experience in using and developing DPRA tools for complex systems. PyCATSHOO is one of the tools developed over last few decades at EDF R&D. PyCATSHOO implements the concept of Piecewise Deterministic Markov Process (PDMP) using distributed stochastic hybrid automata. The principles of this paradigm are described in details in [8].

PyCATSHOO models are used for advanced risk assessment of EDF’s hydro and nuclear electrical generation fleet. PyCATSHOO is grounded on the Object-Oriented (OO) and Multi-Agent System (MAS) paradigms[26]. Following the OO paradigm, PyCATSHOO defines a system, its subsystem or component as a class - an abstract entity that can be instanced into objects. The latter are concrete entities which communicate by message passing and that are able to perform actions on their own encapsulated states. This paradigm has been successfully implemented for modeling and analysis of stochastic hybrid systems as reported in [25]. Following MAS, PyCATSHOO models a system as a collection of objects with a reactive agent-like behavior. A reactive agent acts using a stimulus-response mode where only its current state and its perception of its environment are relevant.

2.1 DPRA Modeling with PyCATSHOO

PyCATSHOO offers a flexible modeling framework that allows for defining generic components (classes) of hybrid stochastic automata to model a given system or a class of systems with a particular behaviour. A hybrid stochastic automaton may exhibit random transitions between its states according to a predefined probability law. It may also exhibit deterministic transitions governed by the evolution of physical parameters.

A modeling process with PyCATSHOO can be summarised as follows:

• •

Conceptual level: A system is decomposed into elementary subsystems, components.

• •

Component level: Each system component is described with a set of hybrid stochastic automata, state variables and message boxes. Message boxes ensure message exchanges between components.

• •

System level: To define the system, the components are instantiated from their corresponding classes. Component message boxes are connected according to the system topology.

A DPRA model in PyCATSHOO combines the characteristics of conceptual model and a software application: it formally describes some aspects of the physical world (for example, a complex mechanical system) for purposes of understanding and communication [28]; it serves a formal basis for further system simulation and analysis.

PyCATSHOO offers Application Programming Interfaces (APIs) in Python and C++ languages. Once the model is designed, the system behaviour is simulated. An analyst needs to use Monte Carlo sampling if the system exhibits random transitions. Sequences (time histories of the system evolution) that lead to desirable end states are traced and clustered.

In [2], various modeling tools for DPRA are discussed. Whereas some modeling tools propose a visual modeling interface, model complexity and high development and maintenance costs are considered the main obstacle for efficient use of DPRA models in industry [10]. Quantitative measures of model maintainability would be of a great value, helping the experts to make better decisions early in the DPRA model development life cycle.

3 Maintainability of Models: State of the Art

ISO 9000 is a set of international standards on quality management. It defines quality as ”the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs” [19]. Maintainability is a quality characteristic that is defined as ”a set of attributes that bear on the effort needed to make specified modifications.”

3.1 Maintainability in Software Engineering

In Standard Glossary of Software Engineering Terminology [20] software maintainability is defined as “the ease with which a software system or component can be modified to correct faults, improve performance or other attributes, or adapt to a changed environment”.

According to ISO/IEC 25010, maintainability is a sub-characteristic of product quality that can be associated with more concrete, ”measurable”, quality metrics. Various types of metrics accepted in SE include: size metrics (e.g., Lines of Code), lexical metrics (e.g., Halstead software science metrics [17]), metrics based on control flow graph (e.g., Mc’Cabe’s cyclomatic complexity [24]) and others.

Metrics specific to Object-Oriented paradigm focus on OO concepts such as object, class, attribute, inheritance, method and message passing. Chidamber & Kemerer’s OO metrics [7] are among the most successful predictors in SE. They include metrics focused on object coupling. In [22] ten software metrics and their impact on the maintainability are studied. The size metrics and McCabe’s complexity are placed among reliable indicators of maintainability. In [31], a systematic review of software maintainability prediction models and metrics is presented. According to this review, a list of successful software maintainability predictors include Halstead metrics, McCabe’s complexity and size metrics.

Abreu’s Metrics for Object-Oriented Design (MOOD) are presented in [1] and evaluated in [18]. According to MOOD, various mechanisms like encapsulation, inheritance, coupling and polymorphism can influence reliability or maintainability of software.

The maintainability index (MI) is a compound metric [30] that helps to determine how easy it will be to maintain a particular body of code. MI uses the Halstead Volume, Cyclomatic Complexity, Total source lines of code.

The models and metrics above address the maintainability at later phases of software development life cycle. In the next part of this section, we consider maintainability at design phase. In particular, maintainability of conceptual models.

3.2 Maintainability in Conceptual Modeling

Conceptual modeling is the activity of formally describing some aspects of the physical and social world around us for the purposes of understanding and communication [28]. Though, in practice, conceptual models are widely used in design and development of IS, they also play an important role in the overall system development life cycle and can be evaluated as representations of the ”real world”. We adopt the definition from [27], where conceptual modelling is defined as a design discipline and conceptual models are considered as ”design artifacts used to actively construct the world rather than simply describe it”. While ISO/IEC 25010 family standards is widely accepted for evaluating software systems, no equivalent standard for evaluating quality of conceptual models exist.

In [27, 29, 6] frameworks for conceptual modeling quality are presented. In [29], the empirical quality is considered as a good indicator of the maintainability. In [6] maintainability of conceptual schema is defined as ”the ease with which the conceptual schema can evolve”. The maintainability of a conceptual schema implies the study of modeling elements cohesion and can be related to the modifiability sub-characteristic from ISO/IEC 25010. Quantitative analysis and estimation of conceptual model quality remains challenging due to lack of measurement [27].

An important body of knowledge is developed adopting and extending the metrics from software engineering to the conceptual modeling. These metrics are used to estimate quality of conceptual models and UML diagrams in particular. In [15], a survey of metrics for UML class diagrams is presented. In [23], a suite of metrics for UML use case diagrams and complexity of UML class diagrams is proposed. Directly measurable metrics such as number of usecases, actors, classes, subclasses, dependencies e.t.c., are used as an early estimate of development efforts, implementation time and cost of the system under development. In [13, 11, 14], a set of metrics to measure structural complexity of UML class diagram are proposed and validated. The authors promote an idea that the structural properties (such as structural complexity and size) of a UML class diagram have an effect on its cognitive complexity. High cognitive complexity reduce model understandability, modifiability and affect its maintainability by the consequence. The authors propose a set of metrics based on OO design state of the art. The experimental work demonstrates that the good indicators of class diagram maintainability include: number of associations, aggregations, generalization, total number of aggregation and generalization hierarchies, the maximum value of direct inheritance tree. In [12] the same group of researchers proposes metrics for measuring complexity of UML statechart diagrams. The authors show that number of states, transitions and activities metrics are correlated with the understandability.

In [32], a multivariate linear model ‘Maintainability Estimation Model for Object-Oriented software in Design phase’ (MEMOOD) is presented. This model estimates the maintainability of class diagrams in terms of their understandability and modifiability. Modifiability is evaluated using the number of classes, generalisations, aggregation hierarchies, generalization hierarchies, direct inheritance tree.

4 Maintainability Assessment of PyCATSHOO Models

Intrinsically, maintainability is associated with the maintenance process, which represents the majority of the costs of a software development life-cycle [31]. It is valid for the model development life cycle as well.

Assessment of conceptual model maintainability can help designers to anticipate model complexity, to incorporate required enhancements and to improve consequently the maintainability of the final software [32]. In this work we adapt and apply several metrics from SE and OO design to evaluate the maintainability of PyCATSHOO models early at the model development life cycle.

Adapting a maintainability definition from [20], we define maintainability of a DPRA model as the ease with which a model or its component can be modified to correct faults, improve performance or other attributes, or adapt to a changed environment.

According to [21], maintainability consists of five sub-characteristics: modularity, reusability, analyzability, modifiability and testability. Modifiability sub-characteristic is most relevant for DPRA models; it specifies a degree to which a product (a DPRA model in our case) can be effectively and efficiently modified without introducing defects.

Similar to software system maintainability, various types of maintainance for DPRA models can be identified:

• •

Adaptive - modifying the model to cope with changes in the environment;

• •

Perfective – improving or enhancing a model to improve overall performance;

• •

Corrective – diagnosing and fixing errors, possibly ones found by users;

• •

Preventive – increasing maintainability or reliability to prevent problems in the future (i.e., model architecture, design).

In this work, we focus on adaptive maintenance that reflects a transformation of simplified DPRA models to real size models.

4.1 Adaptive Maintainability in PyCATSHOO Models

Different classes of modifications can be introduced into a PyCATSHOO model while adapting it to a real size model. In this work, we consider two classes of PyCATSHOO model modifications:

Component level modifications - modifications that consist in adapting structure and/or behavior of a model component (e.g., state variables, PDMP equation methods for continuous variables, start/stop conditions for PDMP controller, transition conditions, message boxes etc.). 2. 2.

System level modifications - modifications that consist in adapting structure and topology of the system (e.g., number of component instances, their parameters, dependencies, connections via message boxes etc.).

Each modification class can be related to different requirements and consequently different technical solutions [16]. We argue that the ”right” architectural and design decisions made for a simplified DPRA model pay off, improving maintainability and reducing complexity of the real-size DPRA model and PyCATSHOO models in particular. The maintainability metrics can serve as indicators for DPRA domain experts in order to assess their design and architectural decisions early in the modeling.

4.2 Selecting Maintainability Metrics

After a review of some existing metrics focusing on maintainability, we select the set of metrics applicable to DPRA models and PyCATSHOO models in particular. We propose a new metric to measure relative modifications - RLOC. We summarize the selected metrics in Table 2. Possibility of seamless integration of metrics into current modeling process with PyCATSHOO and availability of measurement tools are important criteria for our metrics selection. Radon111http://radon.readthedocs.io/en/latest/intro.html is a Python tool which computes various code metrics. Radon supports raw metrics, Cyclomatic Complexity, Halstead metrics and the Maintainability Index. Cloc222https://sourceforge.net/projects/cloc/?source=typ_redirect - Count Lines of Code - is a tool that counts blank lines, comment lines, and physical lines of source code in many programming languages. Given two versions of a code base, cloc can compute differences in source lines.

4.3 Maintainability Assessment Experiment

The goal of our experiment is to validate or refute the following hypothesis:

• •

*H1: *The selected Maintainability metrics will show the difference between original and alternative model designs with respect to applied system level modifications;

• •

*H2: *The selected Maintainability metrics will show the difference between original and alternative model designs with respect to applied component level modifications;

We compare two designs of the Heated Room model reported in [8]:

Set 1: Original design. We take the original system model (Case 0) and create new versions of this model applying system level modifications (Case 1) and component level modifications (Case 2) defined above (see Table 1). These modifications illustrate an adaptive maintenance of a DPRA model in order to reflect real-size system requirements. This set of three models is created following the original design ideas from [8] where system components are connected via PyCATSHOO message boxes, i.e., point-to-point.

Set 2: Alternative design. We create a set of semantically equivalent models of the Heated Room with alternative design (Case 0a - 2a). We promote the low coupling design principle by implementing well-known patterns of object-oriented design (i.e., Mediator, Observer) [33]).

Maintainability Assessment and Analysis of Results. We use the metrics from Table 2 for both sets of models from Table 1 and analyse the results.

In the following sections, we explain the Heated Room test case, provide the details on this experiment and discuss the maitainability assessment results.

5 Case Study: Heated Room

“Heated Room” is the test case reported in [4, 5]. This case is about a room which contains a heater device equipped with a thermostat. The latter switches the heater off when the room temperature reaches 22∘C and switches it on when the room temperature falls below 15∘C. In this situation, a constant heat flow enters the room. The outside temperature is 13∘C and at the initial time t = 0 the room temperature is 17∘C. The flow of the heat leak through the walls is proportional to the difference between inside and outside temperatures. The temperature is governed by a linear differential equation as: $dT/dt=\alpha T+\beta$ where $\alpha$ and $\beta$ depend on the mode of heater. The heater is assumed to have a constant failure rate $\lambda=0.01$ and repair rate $\mu=0.1$.

5.1 Set 1: The Original PyCATSHOO Model of Heated Room

The original model of Heated Room is presented in [8]. It specifies the system components and their representing classes. The diagram in Fig. 1 shows the model from [8]. For the moment of this publication, the PyCATSHOO modeling tool does not have an explicit graphical modeling notation. We use the one that is adopted by EDF experts working with PyCATSHOO.

The Heater class defines two automata that describe functional and dysfunctional behavior of the heater. The OK/NOK transitions are stochastic: the time instants when the heater can fail and can be repaired are distributed according to the exponential distribution law with the parameters $\lambda$ and $\mu$ accordingly. The ON/OFF transitions follow the control logic: the heater switches ON if it is in the functional state (OK) and when the room temperature drops below its specified min. temperature $T_{min}=15$ and switches OFF when it raises above the specified max. temperature $T_{max}=22$. The listing below describes the Heater class, its variables, automata and message boxes according to Fig. 1

The Room class represents an observed subject: its temperature continuously evolves. The Heater and the Room communicate via message boxes: the Room component sends its current temperature to the heater, whereas the Heater sends its current state (ON or OFF) and its power value to the Room.

In the original model design, the Room class contains the specification of a PDMP controller that implements the physics of the process - the evolution of the room temperature over time $T(t)$.

The PDMP controller is a part of the system. Other system components define the functioning of the PDMP controller in a distributed manner: equationMethod() for the room temperature is defined by the Room class, stop conditions stopPDMP() (e.g., when the boundary temperature value is reached) are defined by the Heater class.

In the initial model (Case 0), the Heated Room System class specifies the system with one heater and one room objects connected via corresponding message boxes (mb_Room and mb_Heater).

Fig.2a illustrates the execution trace of the model with the PyCATSHOO simulator: the graph of temperature evolution over time and the heating regime of the heater (ON/OFF). We do not provide the Python code listing due to the space limitations. The detailed specification of the model is presented in [8].

5.2 Adaptive Maintenance of the Heated Room Model: Case 1, 2

System level modifications: Case 1 illustrates the system level modifications. We modify the structure and topology of the Heated Room system by instantiating four heaters (H0..H3) and connecting them explicitly to the room via message boxes.

The Heater class does not change: the heater components independently heat the room following their initial control logic. Several heaters can be ON or OFF at the same time, based on the room temperature and their functioning state (OK or NOK).

We modify the Room class and generalize the PDMP equation method in order to establish the connection between the room and multiple heaters.

The Heated Room System class specifies the system with four heaters and one room connected via corresponding message boxes (mb_Room and mb_Heater).

Component level modifications: Case 2 illustrates the component level modifications. We modify the control logic of the Heater component in order to support the standby redundancy mechanism: one heater (H0) will be declared as the main heater (the highest priority), whereas the other heaters will serve as its backups. Backup heaters are switching on only when all the other heaters with higher priority fail (NOK) and the temperature drops below $T_{min}$. If a heater with higher priority is repaired (OK), the backup heater with lower priority switches off. The following modifications are integrated into the Heater class:

• We add a variable for specifying heater’s priority that will be communicated to other heaters and a corresponding message box:

• We add a variable and a corresponding message box to communicate with the other heaters (i.e., to receive their priority and status)

⬇

1…

2self.pi_priorityOther = self.addReference(”OtherHeatersPriority”)

3self.addMessageBox(”mb_OtherH_I”)

4self.addMessageBoxImport(”mb_OtherH_I”, self.pi_priorityOther,”heaterPr”)

• We modify the ON/OFF and OFF/ON transitions’ conditions taking into account the priority of a current heaters and the statuses/priorities of the other heaters. For example, a heater H with priority x switches on only if all the heaters with priority $y>x$ are off and the room temperature is below the threshold; conversely, the heater H with priority x switches off once a heater with priority $y>x$ is switching on or once the room temperature is above the threshold.

• We modify the stop condition for the PDMP controller taking into account the priority of a current heaters and the statuses/priorities of the other heaters333We omit the details of the getMyPriority(),* getOtherPriority()* methods as the standby redundancy can be implemented in various ways and discussion of a particular algorithm is out of our scope..

We modify the Heated Room System class where new heaters are explicitly connected to the room and to each other via message boxes.

The graph of temperature evolution over time and the heating regime of the four heaters H0..H3 for Case 2 are shown in Fig.2b.

5.3 Set 2: The Alternative Design of Heated Room

We integrate the low-coupling design principles early in modeling phase in order to anticipate component level and system level modifications. We use well-known design patterns from OO software development [33]. The diagram in Fig. 3 shows the alternative design of the Heated room model.

The Mediator design pattern encapsulates the interactions between system component and reduces direct dependencies between them. Integrating mediator early in the model improves system scalability.

In our design, the Mediator class maintains the lists of active components (heaters in our case) and subjects or passive components (i.e., rooms) as shown in Fig. 3. It mediates the communication between the PDMP controller, the heater(s) and the room(s) replacing the point to point connections via message boxes. Note, that arbitrary number of rooms and heaters per room can be configured with this design.

The mediator contains the configurePDMP() method that allows to ”assemble” the PDMP behavior from the parts defined by the active and passive components (i.e., startPDMP(), stopPDMP(), equationMethod()).

Note that different types of active components can be used once they do not change the type of the differential equation: for example, coolers can be used along with heaters as they only change the heating coefficient.

The Heater class is similar to the original design; the message boxes are removed. The Mediator object updates the heaters with the new value of the room temperature (lines 23, 26-28 in listing 11).

The Room class contains a current temperature variable that is updated by the Mediator component. Compared to the original design, the PDMP controller specification is moved from the Room class to the Mediator class. This allows to decouple the room and the heater.

In the initial model (Case 0a), the Heated Room System class specifies the system with one heater and one room attached to a mediator object. configurePDMP() method of the mediator terminates the system configuration.

Simulation results of the Case 0a are shown in Fig.2a.

5.4 Adaptive Maintenance of the Alternative Model: Case 1a, 2a

System level modifications: Case 1a is an alternative design of the Heated Room system with 4 independent heaters. The Heater, the Room and the Mediator classes are not changed.

In the Heated Room System class new heaters are instantiated and added to the mediator.

Component level modifications: Case 2a implements the standby redundancy algorithm for heaters. We modify the Heater class following the logic described in Case 2 in section 5.2. Similar to the Case 2, we integrate the following modifications into the Heater class:

• We add a variable to specify the priority of the heater;

• We modify the ON/OFF and OFF/ON transitions’ conditions taking into account the priority of a current heaters and the statuses/priorities of the other heaters;

• We modify the stop condition for the PDMP controller.

Compared to the original design, we do not connect heaters via message boxes, but implement the Observer design pattern.

The Observer design pattern allows for implementing specific behavior between a group of system components. It defines a one-to-many dependency between objects and when one system component changes state, all its dependents are notified and updated automatically.

The listing below illustrates a fragment of Observer implementation: we add a sensitive method called notifyFailure() to the OK/NOK transition that executes each time the component fails (i.e., goes to the NOK state). This method notifies all the observers (i.e., heaters with lower priority or backup heaters) of the current heater that it fails and that they can now take on the heating. The takeON variable of the corresponding observers is updated. Along those lines, the observers are notified about the reparation or the switching on of their observable heater. We omit the logic of the algorithm.

In the Heated Room System class new heaters are instantiated, grouped according the main/backup topology and added to the mediator:

The graphs of temperature evolution over time and the heating regime of the four heaters H0..H3 for Case 2a are shown in Fig.2b.

6 Maintainability Assessment: Results

We test the hypothesis from section 4.3 applying the metrics to the model sets representing the original and the alternative designs. In this paper, we report on application the following metrics: Line of Code (LOC); RLOC (relative modifications), Cyclomatic Complexity (CC); Halstead’s volume, difficulty, vocabulary, effort, estimated bugs (V, D, $\eta$, E, B); compound measure Maintainability index (MI). These metrics are computed statically from the code.

6.1 Size Metrics

6.1.1 LOC

(Lines of Code) is a software metric used to measure the size of a computer program. It is recognised as a valid indicator of complexity and maintainability. We use LOC as a measure of the size of the PyCATSHOO model. The results of LOC measurement for two sets of PyCATSHOO models (Case 0-2, Case 0a-2a) are shown in Fig.4(a). Stacked columns illustrate LOC per case; various colors correspond to various components. The following can be observed:

• the Mediator class in Set 2 (alternative design) doubles the size of the simplified model;

• For the Set 1, the size of all the classes is growing in response to system and component level modifications;

• For the Set 2, only the Heater and the Heated Room System classes are growing.

Fig. 4(b) shows the evolution of total LOC (model size) per case. Whereas for the initial model the alternative design results in a bigger model, Case 2 and Case 2a models show almost equivalent size.

6.1.2 RLOC.

We propose to measure relative modifications (RLOC) in response to system and component modifications. We use cloc tool to measure the difference between pairs of models (Case 0 and Case 1; Case 1 and Case 2) in both model sets. We calculate total modification as a sum of modified, added and removed lines in the ”adapted” model. We define RLOC for a case as follows:

[TABLE]

This metric allows for more precise measurement of modifications as it takes into consideration not only the total change in model size. Removing or modifying the code (as well as model elements in concept model) are also considered. Table 3 summarises the RLOC measures. RLOC for the Set 1 shows that the model was modified on 82.25% to implement scalability (from 1 to 4 heaters) and further on 108.69% to implement standby redundancy of the heaters.

While having bigger initial size (Fig. 4(a)), the model in the Set 2 was modified on 18.57% to scale for 4 heaters and on 83.4% to implement standby redundancy.

6.2 Cyclomatic Complexity

McCabe [24] proposed Cyclomatic Complexity Measure to quantify complexity of a given software based on its flow-graph. A flow graph is based on decision-making constructs of a program. Fig.5 illustrates cyclomatic complexity measure for the model sets. Whereas the absolute CC values for all models indicate low complexity (below 10), we are interested in the CC evolution in response to adaptive model modifications. The graph in Fig.5 indicates the faster complexity growth for the Set 1. This corroborates with the previous measures.

6.3 Halstead lexical measures

Halstead metrics consider a program as sequence of operators and their associated operands. For a given problem, let:

$\eta_{1}$ = the number of distinct operators;

$\eta_{2}$ = the number of distinct operands;

$N_{1}$ = the total number of operators;

$N_{2}$ = the total number of operands.

From these numbers, several measures can be calculated. In this work, we studied Program length $N$, vocabulary $\eta$, volume $V$, difficulty $D$, effort $E$ and estimated number of errors (or bugs) $B$:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Fig.6(e) illustrates some of Halstead metrics for two modeling sets and an evolution of these metrics in response to adaptive model modifications.

Vocabulary measure is calculated as a sum of distinct operands and operators and can be related to the model complexity. Intuitively, a model with a bigger vocabulary will need more adaptive modifications in response to changed requirements. Thus, it is more difficult to maintain.

The volume measure illustrated in Fig.6(b) shows the evolution of model volume between cases. Note that the results do not correspond to the evolution of model size measured with LOC in Fig. 4(b). It can be explained by the fact that the volume measure takes into account meaningful operands rather lines of code.

The difficulty measure can be related to the difficulty of the PyCATSHOO model to write or to understand. The effort measure can be translated to an estimated development time and thus can indicate the cost of model for a PyCATSHOO model. Delivered bugs is an estimate for the number of errors. For the Case 0a, an estimate number of errors is higher due to implementation of the Mediator; however it seems to remain stable after adaptive modifications (Case 1a, Case 2a). Idem for the difficulty and effort measures.

6.4 Maintainability Index

Maintainability Index is a software metric which measures how maintainable (easy to support and change) the source code is. The maintainability index is calculated as a factored formula consisting of Lines Of Code, Cyclomatic Complexity and Halstead volume:

[TABLE]

Where V is a Halstead Volume; CC - Cyclomatic Complexity; LOC = count of source Lines Of Code. Fig.7 illustrates MI measure for the model sets:

-Room class of the Set 2 and Heated Room System class of both sets are considered as 100% maintainable.

-MI of the Heater class decreases for both sets. This can be explained by its growing complexity and size.

-Average MI, though it remains very high for both sets, drops from 84.2 to 79.2 for the Set 1 and from 86.4 to 85.75 for the Set 2.

7 Conclusion

In this work, we proposed a pragmatic approach for assessing maintainability of DPRA models created with PyCATSHOO based on measures accepted in SE and OO design (LOC, CC, Halstead, MI). We claim that the selected metrics can be used as early indicators for estimating complexity, modifiability and, as a consequence, maintainability of the PyCATSHOO models. To validate our hypothesis, we compare two designs of the ”Heated Room” system.

We create two model sets, each consisting of three models: the initial model with 1 heater and 1 room and the two models, where we apply system level and component level modifications respectively. Models in Set 1 follow the original design reported in [8] and based on point to point connection between system components. Set 2 promotes the low-coupling design principles. We use the selected metrics on both sets of models and analyse the results. We also define our own metric - RLOC - that provides more accurate measure of model modification compared to LOC.

The results seem to validate our hypothesis: the selected maintainability metrics show the difference between original and alternative model designs at early model development phase and thus can help the experts to make design decisions.

Our results demonstrate the following:

The original PyCATSHOO model of Heated room uses direct (point-to-point) connection between its components. Whereas this makes the model clear and easy to create for the case of two components, its complexity grows faster when the number of components increases or when new types of dependencies between components are introduced.

The PyCATSHOO model based on design patterns requires more efforts at creation. In particular, we make some assumptions about how the system will evolve (its size, the types of dependencies) and propose some anticipated design solutions that ensure scalability along the considered dependencies ”by design”. this can be considered as an ”upfront investment” into the maintainability.The evaluation indicates that the low coupling design principle applied early in the modeling improves scalability and maintainability of the real-size model for DPRA. So, the upfront investment pays off.

7.1 Directions for the Future Work

Despite the encouraging results, we consider them as preliminaries. We consider the following directions for our future work:

• •

We plan to replicate our experiment and to assess the maintainability of the real PyCATSHOO projects. The proposed set of measures can be applied to available libraries of PyCATSHOO components (so called PyCATSHOO Knowledge Base or PKB). The results can be used to estimate the current complexity and maintainability of the models and to give an insight about the current design decisions and best practices adopted at EDF.

• •

We plan to explore the executable nature of the PyCATSHOO models and to integrate the metrics based on call graph, similar to [3]. The metrics based on call graph topology can be used to estimate the model evolution and performance.

• •

We plan to extend the list of metrics integrating the metrics on Structural complexity and other OO-specific metrics. The metrics discussed in [32, 15] are of a great interest and can complement the software measures studied in this work. For the initial PyCATSHOO models or toy examples, the number of classes, dependencies and the size of hierarchies are small; as a result of adaptive maintenance and a transformation to a real-size model, the number and variety of components and dependencies between them grow bigger - here the specific OO-metrics can indicate the growth in size and complexity the conceptual model of a system, provide an insight on the element cohesion, coupling and help assessing the models understandability.