Debunking the Myth that Upfront Requirements are Infeasible for Scientific Computing Software

TL;DR

This paper demonstrates that upfront requirements in scientific computing software are feasible and beneficial when using specific principles, templates, and tools, challenging the common belief of their infeasibility.

Contribution

It introduces a practical approach combining perspectives, principles, templates, and tools to make upfront requirements feasible and beneficial in scientific computing software development.

Findings

Upfront requirements are feasible with the proposed approach.

Using these methods improves communication and early error detection.

The approach enhances design decisions and reproducibility.

Abstract

Many in the Scientific Computing Software community believe that upfront requirements are impossible, or at least infeasible. This paper shows requirements are feasible with the following: i) an appropriate perspective ("faking" the final documentation as if requirements were correct and complete from the start, and gathering requirements as if for a family of programs); ii) the aid of the right principles (abstraction, separation of concerns, anticipation of change, and generality); iii) employing SCS specific templates (for Software Requirements and Module Interface Specification); iv) using a design process that enables change (information hiding); and, v) the aid of modern tools (version control, issue tracking, checking, generation and automation tools). Not only are upfront requirements feasible, they provide significant benefits, including facilitating communication, early identification of errors, better design decisions and enabling replicability. The topics listed above are explained, justified and illustrated via an example of software developed by a small team of software and mechanical engineers for modelling the solidification of a metal alloy.

Full text

i) an appropriate perspective (‘faking’ the final documentation as if requirements were correct and complete from the start, and gathering requirements as if for a family of programs);

ii) the aid of the right principles (abstraction, separation of concerns, anticipation of change, and generality);

iii) employing SCS specific templates (for Software Requirements and Module Interface Specification);

iv) using a design process that enables change (information hiding); and,

v) the aid of modern tools (version control, issue tracking, checking, generation and automation tools).

Not only are upfront requirements feasible, they provide significant benefits, including facilitating communication, early identification of errors, better design decisions and enabling replicability. The topics listed above are explained, justified and illustrated via an example of software developed by a small team of software and mechanical engineers for modelling the solidification of a metal alloy.

**keywords: ** software engineering; scientific computing; requirements analysis; information hiding; documentation; casting

††footnotetext: Presented at SE4Science 2019 (ICSE Workshop), https://se4science.org/workshops/se4science19/

1 Introduction

Upfront requirements are often considered infeasible for Scientific Computing Software (SCS). The truth is that requirements for SCS are no more challenging than for any other domain – requirements are difficult for everyone. What is needed is not excuses, but careful, rigorous, sometimes painful thought and effort, as early and as often as possible. With a change in attitude and perspective, the right tools, templates and principles, upfront requirements are possible for SCS.

Throughout this paper, the term upfront requirements does not literally mean requirements documentation that is entirely complete and correct as the first activity in the development process. Rather, upfront requirements means that requirements are considered early in the design process and the documentation is ‘faked’ [1] as if requirements were the first step.

Requirements are documented in a Software Requirements Specification (SRS). An SRS describes the functionalities, expected performance, goals, context, design constraints, external interfaces, and other quality attributes of the software [2]. In a scientific context, an SRS records the necessary terminology, notations, symbol definitions, units, sign conventions, physical system descriptions, goals, assumptions, theoretical models, data definitions, instance models and data constraints [3]. An SRS provides many advantages during software development [1]. For instance, an SRS acts as an official statement of the system requirements for the developers, stakeholders and the end-users, and creating the SRS allows for earlier identification of errors and omissions. As Boehm’s data shows [4], early identification of errors implies a significant return on investment, since the cost of fixing errors increases dramatically for later development stages. An SRS is invaluable for verification, since the quality of software cannot be assessed without a standard against which to judge it. Better design decisions are facilitated by the information captured in an SRS.

Despite these benefits, an SRS is rarely written for SCS software. As the following quotes highlight, previous research has repeatedly shown that many in the community believe that upfront requirements are infeasible for SCS:

• •

“Full up-front requirement specifications are impossible: requirements emerge as the software and the concomitant understanding of the domain progress.” [5]

• •

“Since scientific software is deeply embedded into an exploratory process, you never know where its development might take you. Thus, it is hard to specify the requirements for this kind of software up front as demanded by traditional software processes.” [6]

• •

“The research scientists … do not appreciate the need to articulate requirements fully and upfront as demanded by a staged methodology, and found this articulation very difficult to do.” [7]

• •

“computational scientists generally adopt an agile development approach because they generally do not know the requirements up front.” [8]

• •

“Supplying requirements upfront ran counter to the [scientists] previous experience of developing their own software in the laboratory.” [9]

• •

“None of our interviewees created an up-front formal requirements specification. If regulations in their field mandated a requirements document, they wrote it when the software was almost complete.” [10]

Why is the negative attitude toward upfront requirements, and requirements in general, so prevalent? Part of the reasoning may be that SCS practitioners rarely have formal training in Software Engineering (SE) [11]. Moreover, experiments with applying SE to SCS have often made the mistake of not tailoring the approach to scientists. SCS developers have historically not documented requirements, but that does not mean that they should not. We cannot ignore the possibility that requirements are feasible with the right principles, techniques and tools. Given the benefits of requirements, investigating their feasibility is worth further effort.

This effort is justified because of quality concerns. Faulk et al. [12] observe, “growing concern about the reliability of scientific results based on … software.” Embarrassing failures have occurred, like a retraction of derived molecular protein structures [13], false reproduction of sonoluminescent fusion [14], and fixing and then reintroducing the same error in a large code base three times in 20 year [15]. A recent report on directions for SCS research and education states: “While the volume and complexity of [SCS] have grown substantially … [SCS] traditionally has not received the focused attention it so desperately needs … to fulfill this key role as a cornerstone of long-term collaboration and scientific progress” [16]. Estimates suggest that the number of released faults per thousand executable lines of code during a given program’s life cycle is at best 0.1, and more likely 10 to 100 times worse [17].

The sections of this paper present arguments for the value and feasibility of upfront requirements for SCS. To lend weight to the arguments, a specific real-world software development project is cited. The example consists of software to model the solidification of molten metal alloys [18]. An overview is given in Section 2. The arguments in favour of upfront requirements starts with the previously mentioned idea of ‘faking’ a rational development process (Section 3). Following this, the specific principles and templates for writing an SRS tailored to the needs of SCS are presented in Section 4. For upfront requirements to be effective in an environment where those requirements are constantly changing and evolving, the approach to design also needs to accommodate change. Application of the principles of design for change, along with examples, is discussed in Section 5. Experience has shown that principles and techniques are not enough to lead to a change in software development; therefore, tools that facilitate upfront requirements are summarized in Section 6.

2 Software for Solidification

The running example throughout this paper is called SFS: Software For Solidification [18]. This software is used to analyze solidification data for molten metal alloys. SFS was developed by this paper’s authors as a partnership between software and mechanical engineers.

SFS is typical of SCS for two reasons:

1. the motivation is from a practical scientific/engineering challenge – in this case, how to reduce the number of defective cast metal parts; and,

2. the high level goal (using experimentally measured temperature history data to understand the effect of temperature and temperature gradients on the solid fraction) was clear from the outset, but the details on the appropriate assumptions and numerical techniques were initially uncertain.

The main input for the software consists of temperature history data at different heights within a cylindrical sand mould containing a molten metal alloy. The dimensions of the cylinder are selected to ensure approximately unidirectional heat removal. The experimental setup is shown in Figure 1. The thermocouples for recording temperatures are represented by the label $T_{i}$, where $i$ is the thermocouple number. A water jet is directed at the bottom of the cylinder to aid in the cooling process and to facilitate 1D heat transfer.

During solidification, heat is given out from the liquid phase to form a solid. During this process, the alloy undergoes a phase transition from liquid to 2 phase (solid + liquid) and finally to solid. Typical temperature data for the sequence of thermocouples is shown in Figure 2. The numbering of the thermocouples increases as the distance from the bottom of the cylinder increases. For any instant of time, the thermocouples with the higher temperatures and slower rates of cooling are those nearer the top of the cylinder.

The output of the software is the fraction solid, either as a function of time, temperature or cooling rate. The fraction solid ranges from 0 (liquid) to 1 (solid). The fraction solid is obtained by solving an Ordinary Differential Equation (ODE) based on the temperature data, material properties and processing conditions [18].

3 Faking a Rational Design Process

‘Upfront requirements’ are (almost always) impossible, if the term means software development has to start with a complete and correct statement of requirements. However, ‘upfront requirements’ are possible, if the definition is softened to mean requirements documentation is started as early as possible, and the documentation is continuously maintained, as if a rational design process has been followed [1]. The almost impossibility of having perfect information at the start of a project does not justify not writing requirements. Eventually we will understand the problem at hand – the documentation should be (re)written from that perspective.

An argument against documentation is that developers consider reports for each stage of software development as counterproductive [19, p. 373]. However, reports are only counterproductive if the development process has to follow the same rational process implied by the documentation. In SCS (and other software domains for that matter) frequent change is inevitable. As Parnas and Clements [1] point out, the most logical way to present the documentation is still to ‘fake’ a rational design process. “Software manufacturers can define their own internal process as long as they can effectively map their products onto the ones that the much simpler, faked process requires” [20]. Reusability and maintainability are important qualities for scientific software. Documentation that follows a faked rationale design process is easier to maintain and reuse because the documentation is understandable and standardized. Understandability is improved because the faked documentation only includes the ‘best’ version of any artifacts, with no need to incorporate confusing details around the history of their discovery [1]. Standardization on any process, with a rational process being a logical choice as a standard, facilitates design reviews, change management and the transfer (and modification) of ideas and software between projects [1].

The rational design process promoted here (Figure 3), and employed for SFS, is a variation on the waterfall model described by Parnas and Clements [1]. This V-model variation has a testing related phase associated with each step in the typical waterfall process. The steps include writing requirements in a Software Requirements Specification (SRS), a software architecture in the Module Guide (MG), and the detailed design in the Module Interface Specification (MIS) [21, 22]. Each of these steps has a corresponding Verification and Validation (VnV) plan and associated report.

Upfront requirements are possible if the documentation is maintained from this ‘faking it’ perspective. However, the documentation will only be feasible if the requirements (Section 4) and design documentation (Section 5) facilitate change. Tool support (Section 6) will also be critical, or the frequent changes will be overwhelming. For the SFS example, we started with the SRS, then proceeded to the code, and then refined the requirements and began the design documentation. Following this first “pass,” the focus for subsequent development switched frequently between documents, code and testing. However, the documentation was always maintained as if all of the steps occurred in a rational order.

4 Requirements for Change

Upfront requirements can be feasible with a template tailored to the needs of SCS (Section 4.1) and the use of the following principles: Abstraction (Section 4.2), Separation of Concerns (Section 4.3), Anticipation of Change (Section 4.4), Generality (Section 4.5) and Replicability (Section 4.6). The first four principles come from SE [23], while the last is the cornerstone of the scientific method.

4.1 SRS Template

Writing an SRS generally starts with a template, which provides guidelines and rules for documenting the requirements. Several existing templates contain suggestions on how to avoid complications and how to achieve qualities such as verifiability, maintainability and reusability [2, 24, 25]. Templates are generally selected to suit the problem domain. A template designed for the needs of SCS [3, 26] was selected for documenting SFS, as illustrated in Figure 4. The recommended template is suitable for science, because of its hierarchical structure, which decomposes abstract goals to concrete instance models, with the support of data definitions, assumptions and terminology. Excerpts from the SRS for SFS will be used in the sections below to illustrate how the template and the writing of the SRS supports the SE and scientific principles that make upfront requirements documentation feasible.

4.2 Abstraction

Abstraction focuses on what is important, while ignoring what is irrelevant [23]. Requirements at the right abstraction level lead to a model where the likely changes are excluded as irrelevant details. With the aid of the principle of abstraction, the SRS for SFS underwent very little modification over the life of the project. Although considerable exploration and experimentation were necessary, this exploration took place through the design and implementation, not the SRS.

The overall goal for SFS (Figure 5) was written so that it would remain true throughout the project. The goal statement does not list specific material properties or initial conditions, since they may change as different models are explored.

The goal statement mentions temperature readings. This concept needs to be further refined to how discrete readings are transformed into a continuous approximation of the evolution of the temperature field over time. This is done through the general definition (GD4) of the transformation of the experimental data (Figure 6). GD4 abstracts the concept of taking experimental data ($\mathit{Tdata}$) at the thermocouples (shown in Figure 2) over time and determining a function $T(y,t)$. This function takes the position $y$ ($\mathrm{m}$), as measured from the bottom of the cylinder, and the time $t$ ($\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}$) and returns the temperature ($\mathrm{\SIUnitSymbolCelsius}$). $\text{fit}()$ is a function that takes the thermocouple data $\mathit{Tdata}$, the locations of the thermocouples $y_{TC}$ and the time step $dt$, and returns the appropriate function $T(y,t)$. $m$ is the number of instants of time where the thermocouple data is measured and $n$ is the number of thermocouples. The specific approach for fitting, such as interpolation versus regression, is not specified, since these details are not relevant at this stage. All that needs to be known is that there will be a function for finding the temperature at any point in space and time.

The function $T(y,t)$ is used to find the solid fraction $f_{S}$, as given in the excerpt of an instance model (Figure 7). The instance models are the closest the SRS gets to code, since they list the program inputs and outputs. Abstraction is still employed though, since the numerical algorithm for finding partial derivatives of $T(y,t)$ is not given, nor is the algorithm for solving the ODE. The cross-references to Data Definitions (DD) in Figure 7 show the interrelation between the SRS parts.

4.3 Separation of Concerns

With the principle of separation of concerns, we reduce a complex problem to a set of simpler problems [23]. This principle was used in the design of the SRS template [26, 3]. For instance, the different sections of the table of contents (Figure 4) can each be considered separately. As an example, thinking about functional requirements, like the governing equations for physical models, is made easier if one momentarily neglects consideration of nonfunctional requirements, like performance and portability.

Upfront requirements become much more stable if the SRS documents the physical models, but does not address numerical methods. The decisions on numerical techniques, such as the ODE solver, should be postponed to the design stage. Knowing the most appropriate numerical technique is difficult at the outset, and the initial choice is likely to change. The physics on the other hand, if written with the right abstraction, is much less likely to change.

Separation of concerns allows us to improve the overall software quality and software developer productivity because it allows us to distribute the human effort and thought throughout the development process. Human nature is to put off difficult tasks. For instance, a developer might prefer to develop code rather than think about how to test their code. However, many testing concerns do not have to wait until the code is written. The associated thought and effort can happen upfront. For instance, the properties of a correct solution (Section 5.vii of the SRS) are often known at the outset of a project. These properties should be documented, and used as the basis for test cases in the System VnV Plan (Figure 3). What the SRS template terms ‘properties of a correct solution’ is often called initial hypotheses and metamorphic tests by others [27].

SFS provides an example of a property of a correct solution for the temperature field $T(y,t)$, which is shown graphically in Figure 2 and described in Figure 6. Physics imposes the property that for any $t$, $t>0$, $(\forall\,y_{1},y_{2}:\mathbb{R}|0\leq y_{1}\leq H\wedge 0\leq y_{2}\leq H\wedge y_{1}<y_{2}:T_{t}(y_{2})\geq T_{t}(y_{1}))$ where $T_{t}=\lambda y:T(y,t)$. That is $T_{t}(y)$ is non-decreasing with increasing $y$ ($\lambda$ is an anonymous function). This property was known at the outset of the project and provided assertions in the unit tests that later detected a mistake in the calibration of the thermocouples.

4.4 Anticipation of Change

Upfront requirements are possible, if we anticipate future changes. The SRS template used for SFS was designed with change in mind [3, 26]. For instance, assumptions are listed with their traceability to other parts of the SRS explicitly indicated. As an example for SFS, one of the assumptions is that the density of the alloy can be expressed as a linear combination of the density values at the beginning and end of the solidification. This assumption maps to the definition of density and the models that use density for their calculations. If this assumption should change, the consequences of this change are clearly indicated. Other aspects of the template that support change include explicit traceability between the information in the SRS, as indicated in Figure 8 (documented by cross-references and a full traceability matrix), a section on likely changes (for instance, the heat transfer at the bottom of the cylinder may later be changed from constant to temperature dependent), and the use of symbolic constants.

The template anticipates change, but successfully using it requires a new perspective where the author(s) think of their task as documenting a family of SCS programs, instead of just one. As for the properties of a correct solution mentioned in Section 4.3, this will mean intense effort/thought at the beginning of the development process. A program family approach, where commonalities are reused and variabilities are identified and systematically handled, is natural for scientific software [28]. The laws of physics are stable; they are almost universally accepted, well understood, and slow to change. At the appropriate abstraction level, many problems have significant commonality, since a large class of physical models are instances of a relatively small number of conservation equations (conservation of energy, mass and momentum). The challenge is to know which simplifying assumptions are appropriate. Therefore, as mentioned above, the assumptions need to be documented clearly, and explicit traceability is required to show which parts of the model they influence. For general purpose SCS, like a solver for a system of linear equations, the program family should generally be clear from the start, since general purpose tools are based on well understood mathematics.

SFS has benefitted from application of the principle of anticipation of change because several sections are straightforward modifications of previous SRS documents. For instance, SFS shares verbatim the abstract theoretical model for conservation of thermal energy with projects on modelling a fuel pin in a nuclear reactor [29] and predicting the temperature of a solar water heating tank (texthttps://github.com/smiths/swhs).

4.5 Generality

To aid future reuse and to facilitate software design and implementation, the SRS should be written with generality. This often involves using abstraction, which is why the abstract theoretical model of conservation of thermal energy can be reused in multiple projects (Section 4.4).

A key example of generality in SFS is found in the decision to write the instance model on the calculation of $f_{S}$ in the standard form of a system of ODEs (Figure 7). Other work related to calculation of $f_{S}$ does not target this standard form [30]. As a consequence, this other work requires development of an ad hoc algorithm for determining the solid fraction. The design and implementation of SFS on the other hand can, because of its generality, take advantage of the wealth of numerical ODE solver libraries available.

4.6 Replicability

Although reproducibility is the cornerstone of the scientific method, until recently it has not been treated seriously in software [31]. Fortunately, in recent years multiple conferences, workshops and individuals are calling for dramatic change [32]. Reproducibility problems are even more extreme when the goal is replicability. A third party should be able to repeat a study using only the description of the methodology from a published article , with no access to the original code or computing environment [31]. However, replicability is rarely achieved, as shown for microarray gene expression [33] and for economics modelling [34]. Replicability is not achieved because journal papers cannot possibly document all definitions, assumptions, models etc. An SRS addresses completeness and ambiguity problems, since a template is followed, there is no page limit and, SE principles are followed. The goal is to capture all of the required knowledge, including derivation of equations and rationales. If the SCS community is serious about replicability, documentation of an SRS at the level of detail described in this paper is required.

5 Design for Change

Upfront requirements, using the principle of ‘faking it,’ are only feasible if the design can be modified in response to changing requirements. This is possible by applying the same principles listed in the previous section. The design of SFS is captured in the Module Interface Specification (MIS). This document follows a previous template [22]. This template is based on a textual design specification [23, Chapter 4] and the mathematical formalism of a module state machine [35].

An important module in SFS is the one responsible for calculating the temperature at any point in space and time by using the temperature history data at the thermocouples, as shown in the SRS under general definition GD4 (Figure 6). The design uses a special technique for anticipation of change: information hiding [36]. The changeable secrets of the modules are hidden behind the stable interface. This means the design decision can be changed, without a need to modify the modules that rely on the provided services.

Separation of concerns is used in the design to handle the complexity of fitting over two dimensions (distance and time). First an Abstract Data Type (FunctT) is introduced to capture the cooling curves, which are shown for each thermocouple in Figure 2. Second an ADT is introduced for the contours (ContrT), which consists of a sequence of cooling curves (FunctT objects), with each curve at a different location. Applying the principle of generality, the modules are not specific for temperature; they can be used for any two dimensional data. For instance, the same approach was used in a program to calculate the risk of glass panes failing (https://github.com/smiths/caseStudies).

The syntax of the access programs for FunctT is given in Figure 9. The MIS template shows application of the principle of separation of concerns, by separating the syntax and semantics. The interface provides a constructor (new FunctT) that takes two vectors of data and returns a FunctT object. Accessors include minD and maxD for finding the extreme limits of the independent data variable. The accessor eval returns the value of the dependent variable ($x$) given a value for the independent variable ($y$). The syntax of FuncT shows exceptions if the data for the independent variable data is not in ascending order, of if the number of data points in the two sequences do not match, or if a function evaluation is sought outside of the domain of the given data.

The corresponding semantics for FunctT is given in Figure 10. The MIS uses abstraction, with the functional fit represented by a state variable with a function type ($\mathbb{R}\rightarrow\mathbb{R}$). The specific function is hidden. To make it easy to change, a local function is defined for interpolation (interp). This function can be changed to any order of interpolation, or it could be changed to regression, or a spline. Defining a new functional fit, just requires redefining interp. Following the principle of information hiding, the interface to FunctT would remain unchanged. This facilitates experimentation, which was necessary for determining the best approach for SFS.

Figure 11 shows the syntax and semantics for the MIS for ContourADT. The full data set is built up by adding each successive thermocouple’s data. The accessors, eval, dydx and d2ydx2 are used to calculate the values of $T(y,t)$, $\frac{\partial T}{\partial t}$ and $\frac{\partial^{2}T}{\partial y^{2}}$, respectively. The access program slice returns a new FuncT that would hold the temperature values through the height of the cylinder for a given value of the dependent variable $t$. In this case the slice will use the same fitting algorithm as for the cooling curves. In the actual MIS for SFS the order of the interpolating polynomial was exposed as an input variable, so that the fit through the height could be a different order than the fit for an individual cooling curve.

6 Tool Support

Experience has shown that principles and templates are not enough for dramatic change in development practices. Although the value of documentation is recognized, there is a sense that writing and maintaining the documentation is too great an effort [37]. The positive influence of tool support is observed in the CRAN (Comprehensive R Archive Network) community, where documentation generation and consistency checking tools allow a single SCS developer to achieve similar software quality to a team of developers [38].

SCS developers should use tools for issue tracking and version control. Issue tracking is considered a central quality assurance process [39]. For SFS, git was used for version control and GitLab for issue tracking. The issue tracker facilitated bringing the team knowledge together, especially when review questions were assigned to the domain experts.

Going forward, a knowledge-based approach for scientific software development holds promise. Ideally, developers should be able to create high quality documentation without the drudgery of writing and maintaining it. A potential solution is to generate the documentation automatically by using Domain Specific Languages (DSLs) over a base of scientific knowledge. DSLs and code/document generation provide a transformative technology for documentation, design and verification [6, 40]. DSLs allow upfront changes to be propagated through a project and they provide checks for consistency and completeness. Moreover, a generative approach removes the maintenance nightmare of documentation duplicates and near duplicates, since knowledge is only captured once and automatically transformed as needed.

7 Concluding Remarks

Without dramatic intervention, our collective confidence SCS is due for a collapse. Successfully building SCS requires communication between software developers and experts from multiple domains. Collaboration is difficult at the best of times, and is made worse because developers avoid the upfront effort of thinking about the requirements and the known properties of a correct solution. Developers tend to favour handcrafted solutions over adapting SE processes, methods and tools [12]. Handcrafted solutions do not account for the inevitable changes in requirements, design and implementation. The twin challenges of changing requirements and inadequate documentation conspire to make computational results notoriously difficult to reproduce, especially in the case of one researcher independently replicating another’s results [40].

The solution to quality problems is applying, adapting and developing SE principles, tools and techniques. However, typical processes are a barrier to progress. “To break the gridlock, we must establish a degree of cooperation and collaboration with the [SE] community that does not yet exist” [12]. “There is a need to improve the transfer of existing practices and tools from … [SE] to [SCS]. In addition, … there is a need for research to specifically develop methods and tools that are tailored to the domain” [41]. “Use of [SE] practices could increase the correctness of scientific software and the efficiency of its development.” [42]. The current paper has highlighted perspectives, SE principles, templates and tools as part of a path toward interdisciplinary SE and SCS.

Acknowledgements

The financial support of the Natural Sciences and Engineering Research Council (NSERC), Automotive Partnership Grant, APC 435504-12 is gratefully acknowledged.