A new method for a priori practical identifiability

TL;DR

This paper introduces a new method for analyzing identifiability in biomedical models by relaxing the assumption that all derivatives of measured variables are available.

Contribution

The paper introduces (ν1,...,νm)-identifiability and new algorithms to assess it without requiring all derivatives.

Findings

The new method allows for identifiability analysis with limited derivative information.

In some models, assuming only up to third-order derivatives reduces identifiable parameters significantly.

Previously assumed identifiability requires at least 20 derivatives of all signals in some models.

Abstract

Background and objective: Practical identifiability analysis, i.e., ascertaining whether a model property can be determined from given data, is central to model-based data analysis in biomedicine. The main approaches used today all require that coverage of the parameter space be exhaustive, which is usually impossible. An alternative could be using structural identifiability methods, since they do not need such coverage. However, current structural methods are unsuited for practical identifiability analysis, since they assume that all higher-order derivatives of the measured variables are available. Herein, we provide new definitions and methods that allow for this assumption to be relaxed. Methods and results: We introduce the concept of (ν1,...,νm)-identifiability, which differs from previous definitions in that it assumes that only the first νi derivatives of the measurement signal yi are available. This new type of identifiability can be determined using our new algorithms, as is demonstrated by applications to various published biomedical models. Our methods allow for identifiability of not only parameters, but of any model property, i.e., observability. These new results provide further strengthening of conclusions made in previous analysis of these models. For the first time, we can quantify the impact of the assumption that all derivatives are available in specific examples. If one, e.g., assumes that only up to third order derivatives, instead of all derivatives, are available, the number of identifiable parameters drops from 17 to 1 for a Drosophila model, and from 21 to 6 for an NF-κB model. In both models, the previously obtained identifiability is present only if at least 20 derivatives of all measurement signals are available. Conclusion: Our results demonstrate that the assumption regarding availability of derivatives made in traditional structural identifiability analysis requires a big overestimation regarding the number of parameters that can be estimated. Our new methods and algorithms allow for this assumption to be relaxed, bringing structural identifiability methodology one step closer to practical identifiability analysis.

Figures

Illustration of the upper limit for how many derivatives can be estimated from noisy data.A: Examples of 10 different realizations of noisy data (colored lines) and the true function y(t)=1+9cos(t) (black line) B: Illustrations of the 1st, 3rd, and 5th derivatives of the function y(t) with examples of a poorly estimated derivative (red line) and an accurately estimated derivative (green line). C: The relationship between the average number of accurately estimated derivatives and the magnitude of the added noise σ/the signal-to-noise ratio. The bars show the mean number of accurately estimated derivatives at each noise level and the error bars show the standard error of the mean (SEM).

Variation of the number of identifiable parameters with maximum number of available output derivatives in a model of Drosophila period protein.The horizontal axis shows k: ∀i(j≤νmax↔yi(j) can be reliably estimated). The vertical axis shows the number of identifiable states and parameters in the model. Blue circles represent the case with only one output: y1=PN. Red circles represent the case with two outputs: y1=PN, y2 = M.

Variation of the number of identifiable parameters with maximum number of available output derivatives in a model of NF-κB regulatory module.The horizontal axis shows νmax: ∀i(j≤νmax↔yi(j) can be reliably estimated). The vertical axis shows the number of identifiable states and parameters in the model.

Equations

Full text

1 Background

Biology and medicine involve the study of a myriad of time-dependent variables, which cross-talk with each other in complex networks. To deal with this complexity, one often makes use of mechanistic mathematical models, often formulated using ordinary differential equations (ODEs). These ODEs represent hypotheses regarding the biological mechanisms that can explain given biological data [9]. The model-based data analysis of such real data revolves around a correct uncertainty analysis of parameters and predictions: to what degree can one determine the parameters and predictions of interest? This uncertainty analysis is formally called identifiability analysis if it deals with parameter uncertainty, and observability analysis if it deals with other model properties such as states and composite variables [10]. Both observability and identifiability analysis can be done in many different ways, but they are generally subdivided into two types of approaches: structural and practical.

Practical identifiability analysis revolves around the specific data one has collected in a specific situation [10]. In other words, practical identifiability analysis looks at the real situation, where the data has noise, and where the experiment might not have been done in an optimal way to generate information about all parameters. Historically, one has often used sensitivity-based methods to study practical identifiability, such as analysis of the Fisher Information Matrix [35], and the covariance of the parameters [11]. In the last 10 years, these methods have been increasingly replaced by Markov Chain Monte Carlo (MCMC) and profile-likelihood based methods [10,11,25]. The benefit of these more modern methods is that they study global properties, and do not have to assume that the system is identifiable to assess identifiability [10,35]. These methods are highly useful and flexible, but they still have limitations. Most importantly, they rely on the exhaustive coverage of the entire parameter space, either using the MCMC sampling or using the optimization step in profile likelihood. For large models, this optimization step is difficult and time-consuming even for an expert in the field [26,30]. This difficulty will always leave the analysis results with an inherent doubt: if the coverage of the parameter space fails, the degree of identifiability (the calculated confidence intervals) is not correct.

Structural identifiability analysis has historically been the mirror image of practical identifiability analysis. Structural methods normally do not take a specific data set or the noise level into account. Instead, structural identifiability analysis only considers the form of the equations, including the measurement equations. Furthermore, structural identifiability analysis does not rely on an exhaustive coverage of the entire parameter space. Instead, structural identifiability analysis makes use of powerful theories from differential algebra and differential geometry to investigate whether there are structural limits for whether a parameter or a model property ever can be determined from a given measurement. Methods have been developed to determine which states and parameters are locally identifiable [29] and globally identifiable [6,15], to discover parameter combinations that are identifiable [19,20,23,24,30], and to suggest modifications in the modeling process based on identifiability [4,14]. In the last two decades, powerful methods have been developed that can do such an analysis also for relatively large biological models, featuring 50 to 100 states and parameters [4].

Because structural identifiability methods now can be applied to real, large models, it would be highly interesting if they also could perform practical identifiability analysis, where the limitations of a specific dataset are taken into account. One prominent such assumption is that current methods assume that all derivatives of all measurement signals can be estimated from the data. In practical situations, with real data, this assumption is never fulfilled. It is therefore a critical flaw that we are lacking definitions and methods to analyze the consequences of this unfulfilled assumption.

In summary, there is a need to develop new methods that combine the strengths of the two approaches: a method that does not rely on overly idealistic assumptions regarding the real situations, but that also does not rely on optimization and coverage of the entire parameter space. However, such methods have not yet been proposed. Herein, we introduce a new type of practical and structural identifiability, which puts an upper limit on how many times each measurement signal can be differentiated. We also introduce new algorithms that can calculate this new type of identifiability, and apply these to a series of published models. For all studied examples, traditional structural identifiability methods have widely overestimated the number of identifiable parameters: with a maximum of 3 derivatives available, more than 90% of the previously identifiable parameters lose their identifiability.

We present a new method for determining structural identifiability in a more realistic setting, where only a given finite number of derivatives are available for each measurement signal. While this is novel, there are a few related methods that should be mentioned. Most structural identifiability methods assume that all derivatives of outputs are measurable and all derivatives of all outputs are sufficiently varying. The latter condition is not always satisfied in practical modeling as, for example, a constant function may be used as an input, and this may lead to incorrect implementation of a model. In [32] a method for determining how many derivatives of the inputs must be non-zero was given, and an implementation was presented in [33]. In [4] an implementation was given for determining minimal sets of outputs that make a system structurally locally identifiable.

2 Results and discussion

2.1 Problem: There is an upper limit for how many derivatives can be estimated from biological data

There is usually an upper limit for the number of derivatives of an output signal that can be estimated from data. This is illustrated in the following example.

Example 2.1. Consider the system

$$[eqn]$$

where x_1_ and x_2_ are state variables, $[eqn]$ is a constant parameter, and y is an observable output variable; that is y is measurable by experiments but x_1_, x_2_ and $[eqn]$ are unknown. Suppose that, unbeknownst to the observer, $[eqn]$ , x_1_(0) = 10, and x_2_(0) = 0. Consequently the true signal of y is given by $[eqn]$ . However, in practice, the measured signal will not show a smooth curve but rather a discrete time series that also has added noise. An illustration of this measurement signal can be found in Fig 1A, where 10 realizations of the function $[eqn]$ , where the second term represents normally distributed noise with mean 0 and standard deviation 2, are depicted alongside the true function y(t) (Fig 1A). The number of derivatives that can accurately be estimated can be evaluated by deriving the function of a polynomial interpolation of these data points and comparing these estimated derivatives to the derivatives of the true function (Fig 1B). More specifically, this example shows a 6^th^ degree polynomial interpolation for t-values ranging from 0 to 6 in increments of 0.5. Subsequently, the 0^th^ to the 5^th^ derivatives of these interpolations are then compared to the derivatives of the true function y(t). The number of derivatives that can accurately be estimated is dependent on the magnitude of noise in the system. More specifically, the magnitude of the added noise increases the number of accurately estimated derivatives declines (Fig 1C). To illustrate this dependency, the example above was repeated with the added noise sigma ranging from 0.1 to 10 in increments of 0.1 ( $[eqn]$ ) and with 1000 data realizations generated at each noise level. Fig 1C shows the average number of derivatives that could accurately be estimated at each noise level. A derivative is considered accurate if the square root of the sum of squared residuals does not exceed the amplitude of y. From this figure, it is clear that the number of derivatives that can accurately be estimated diminishes rapidly as the signal-to-noise ratio approaches 1.

Illustration of the upper limit for how many derivatives can be estimated from noisy data.A: Examples of 10 different realizations of noisy data (colored lines) and the true function y(t)=1+9cos(t) (black line) B: Illustrations of the 1st, 3rd, and 5th derivatives of the function y(t) with examples of a poorly estimated derivative (red line) and an accurately estimated derivative (green line). C: The relationship between the average number of accurately estimated derivatives and the magnitude of the added noise σ/the signal-to-noise ratio. The bars show the mean number of accurately estimated derivatives at each noise level and the error bars show the standard error of the mean (SEM).

The following example illustrates how this lack of availability of all derivatives of y is associated with an unrealistic assessment of whether parameters are identifiable.

Example 2.2. Consider the system

$$[eqn]$$

where x_1_, x_2_, and x_3_ are (unobserved) state variables, $[eqn]$ and $[eqn]$ are constant parameters, and y is an (observed) output variable.

Running the algorithm from [29] shows that x_1_, x_2_, x_3_, $[eqn]$ , and $[eqn]$ are locally identifiable. Let us attempt to estimate $[eqn]$ from observation of y. A relation among $[eqn]$ and y over $[eqn]$ is $[eqn]$ . Thus we can estimate $[eqn]$ by using our observed values of $[eqn]$ and $[eqn]$ at some time t_0_:

$$[eqn]$$

In practice, the measured signal y may be too noisy to give an accurate estimation of $[eqn]$ . In such a situation, $[eqn]$ is not identifiable from the given data, even though a structural analysis says that it is. Suppose now that we know that our measurements are good enough to give reliable estimates of $[eqn]$ but of no higher derivative of y. To use structural methods to determine the degree of identifiability in such a situation, we need new definitions and methods.

Remark 2.3. Our approach assumes measurements of several derivatives of the output signal are made at a single time. In practice it is more common to measure the 0-th derivative of the output signal at several times. Having measurements with some uncertainty of $[eqn]$ is equivalent to having measurements with some uncertainty of $[eqn]$ . Hence this aspect of our approach does not limit its generality.

For Example 2.2, one can estimate $[eqn]$ using the relation $[eqn]$ together with measurements of y and $[eqn]$ at three different times. However knowledge of $[eqn]$ at three times can be used to estimate $[eqn]$ at a single time.

2.2 A new definition for practical and structural identifiability: (ν1,…,νm)-identifiable

We now provide an intuitively straightforward definition that is useful for understanding the new methods. In Sect 5, we provide a precise definition, together with analytical proofs of all stated method properties.

Let $[eqn]$ be a non-negative integer and let n, m, and r be positive integers. Let $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ . Let $[eqn]$ and $[eqn]$ be tuples of rational functions in x, u, and $[eqn]$ over $[eqn]$ . This setup determines a class of systems of ODEs with initial conditions, or a model class:

$$[eqn]$$

where x are state variables, which usually correspond to time-varying concentrations or physical properties; $[eqn]$ are the initial values of the state variables; $[eqn]$ are constant parameters, e.g. corresponding to rate constants or volumes; u are input variables, which are observed and typically controlled by the experimentalist; and y are output variables, which are observed experimentally. With these notations in place, the new definition is roughly given by:

Definition 2.4 (roughly stated). Let $[eqn]$ be non-negative integers. Rational expression $[eqn]$ is said to be $[eqn]$ -identifiable if perfect knowledge of the first $[eqn]$ derivatives of y_i_ at a particular time allows us to determine the value of h at that point in time, up to a finite set.

This definition, which is more technically stated in Definition 5.4, provides the essence of the new definition: given only a limited set of derivatives of the different measurement signals, can we determine the parameter $[eqn]$ to be locally identifiable, i.e. as belonging to a finite set? This definition is analogous to traditional definitions of structural identifiability, with the only addition being that we now only have access to a finite set of derivatives. In other words, in traditional structural identifiability analysis, no such constraints on the number of derivatives available is present. This new definition is closer to practical identifiability since, with real biological data, one can only estimate a few derivatives from the measurement data. Note that in practice it is more common to measure the signal at multiple times. We explain in Remark 2.3 that this does not limit the generality of our approach.

2.3 Solution: Our new algorithms for calculating practical structural identifiability

For most models, it would be difficult to prove the $[eqn]$ -identifiability of a given parameter directly from the definition. We provide Algorithm 1, which can do this using only straightforward algebraic computations. The algorithm applies to any rational combination of states and parameters, of which an individual parameter is a special case.

Algorithm 1 Determine whether a given element of $[eqn]$ is $[eqn]$ -identifiable.

Input : Equations $[eqn]$

non-negative integers $[eqn]$

rational expression $[eqn]$

Output : “Yes” if h is $[eqn]$ -identifiable

“No” if h is not $[eqn]$ -identifiable Step 1: Write the matrix

$$[eqn]$$

Step 2: Compute the rank of J.

Step 3: Write the matrix Jh, obtained by adding the row

$$[eqn]$$

to J. Then calculate the rank of Jh.

If $[eqn]$ , conclude that h is not $[eqn]$ -identifiable. (Output “No”).

If $[eqn]$ , conclude that h is $[eqn]$ -identifiable. (Output “Yes”).

In the case where one is interested in just an individual parameter or state, one can perform Algorithm 2, where one column is removed instead of a row added, resulting in a smaller matrix.

Algorithm 2 Determine whether a given element of $[eqn]$ is $[eqn]$ -identifiable.

Input : Equations $[eqn]$

non-negative integers $[eqn]$

choice of $[eqn]$ (resp. $[eqn]$ )

Output : “Yes” if $[eqn]$ (resp. xi) is $[eqn]$ -identifiable

“No” if $[eqn]$ (resp. xi) is not $[eqn]$ -identifiable Step 1: Write the matrix J as in Algorithm 1.

Step 2: Compute the rank of J.

Step 3: Remove the column corresponding to $[eqn]$ (resp. xi) and compute the rank of the resulting matrix.

If the rank has decreased, conclude that $[eqn]$ (resp. xi) is $[eqn]$ -identifiable. (Output “Yes”)

If the rank is the same, conclude that $[eqn]$ (resp. xi) is not $[eqn]$ -identifiable. (Output “No”)

The precise definition of $[eqn]$ -identifiability is given in Definition 5.4, and we prove that Algorithm 1 and 2 are correct in Sect 5.2.

The symbolic calculations can be impractically slow for all but the smallest models. Therefore we have also provided probabilistic analogs of these algorithms that are much faster in practice. These are presented and their correctness is proven in Sect 5.3.

2.4 Intuitive understanding of the new algorithms

We first discuss Algorithm 1. Each row of J represents how an output, or one of its derivatives, varies with respect to small changes in each state and parameter. If the last row of Jh is a linear combination of the other rows, then the variation of h can be accounted for as a combination of the variations of the output derivatives. This, in turn, means that it seems like h can be expressed in terms of the known signals, i.e. that h is identifiable. Conversely, if the last row of Jh is linearly independent of the other rows, then the variation of h cannot be explained in terms of that of the available derivatives of the outputs, and one should not expect to be able to determine h from the available outputs and their derivatives.

A similar explanation lies behind Algorithm 2. The column corresponding to $[eqn]$ is dependent on the other columns if and only if the variation in the outputs and their derivatives is fully explained by variation in the other parameters and states, which is true if and only if $[eqn]$ cannot be determined from numerical knowledge of the outputs and their derivatives, i.e. $[eqn]$ is not identifiable.

2.5 Examples

2.5.1 Algorithm 1 determines identifiability of rational quantities, which is important in hypothesis testing.

Algorithm 1 (and its faster probabilistic version Algorithm 3) can be used to analyze so-called core predictions (see [10]). A core prediction is a well-determined property, i.e. a model prediction with a small uncertainty. Such core predictions are therefore often tested in future experiments, and are a central part of model-based hypothesis testing. Algorithm 1 can test whether or not such a model property really can be well-determined, given the available data. We illustrate this new possibility in the following two examples.

Example 2.5. The following equations give a model of insulin (ins) binding and activation of insulin receptor (IR). This model was originally presented in [7], and is explained more in detail therein and in the supplementary materials.

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

This model was one of several hypotheses tested in the article. To draw one of the conclusions in that article, the authors looked at the proportion of insulin receptor bound to an internal membrane:

$$[eqn]$$

It was predicted that prop_i_ should be between .55 and .80. Despite the fact that there are four outputs, none of the seven states (insulin receptor or substrate quantities at different locations) are identifiable, so it is not immediately obvious whether prop_i_ can be estimated from the measurements. Algorithm 3 with $[eqn]$ for all i shows that this quantity is indeed single-experiment locally identifiable (see Definition 5.8 and Corollary 5.10). The authors were able to experimentally determine the proportion to be $[eqn]$ . Thus they were able to reject the model, which was the conclusion in that step in the paper.

In summary, our analysis shows that the proportion of receptors that are bound to an internal membrane indeed is an identifiable model property. This new result holds from a structural point of view, without assuming that all derivatives are known. These results also show this without relying on optimization or exhaustive parameter sampling, as is done using the traditional practical identifiability analysis done in the original papers [7,10]. This new result thus strengthens and confirms the conclusion drawn in the original paper.

Example 2.6. The equations below constitute the model of the interaction between liver and pancreatic cells from [8].

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

There are seven states, $[eqn]$ , two outputs, $[eqn]$ , and no inputs. Numerical values were inserted for parameters whose values were known from operating specifications or estimated from the literature. The remaining parameters, here denoted by $[eqn]$ , were to be determined from experimental data. One quantity of interest was

$$[eqn]$$

representing hepatic insulin sensitivity (see [8, Fig. 4 Plot E]). Algorithm 3 tells us that this quantity is indeed identifiable. Moreover, by using different values of $[eqn]$ we find that it is (6,5)-identifiable.

2.5.2 Identifiability tends to increase with increased order of observability.

It is typically the case that as we increase the number of derivatives that can be reliably estimated, the number of identifiable parameters increases, often dramatically. Therefore assuming that all derivatives are available can give a large overestimate of the number of identifiable parameters. How large this overestimate is can now be determined by our new algorithms. We illustrate this capability in the following two examples.

Example 2.7 (Drosophila period protein). The following equations constitute the model that was used in [13] to model Drosophila period protein.

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

A local structural identifiability analysis done in [29, p. 737] shows that all states and parameters other than M, $[eqn]$ , $[eqn]$ , K_m_, and k_s_ are identifiable. However, Algorithm 4 shows that only P_N_ is identifiable using 19 or fewer derivatives of y_1_. If in addition we observe M, it was shown in [29] that all states and parameters are identifiable. However Algorithm 4 shows this is the case only if at least 16 derivatives of the outputs are available. Fig 2 shows the number of states and parameters that are identifiable as a function of the maximum number of derivatives of each output that are available; for example $[eqn]$ means that exactly $[eqn]$ are available for all i.

Variation of the number of identifiable parameters with maximum number of available output derivatives in a model of Drosophila period protein.The horizontal axis shows k: ∀i(j≤νmax↔yi(j) can be reliably estimated). The vertical axis shows the number of identifiable states and parameters in the model. Blue circles represent the case with only one output: y1=PN. Red circles represent the case with two outputs: y1=PN, y2 = M.

Blue circles represent output set $[eqn]$ and red circles represent output set $[eqn]$ .

Example 2.8 (NF- $[eqn]$ B). We consider a model of NF- $[eqn]$ B regulatory module, as presented below. This model was first introduced in [18], and an identifiability analysis was done in [17]. The model has 15 states, 28 parameters, 4 outputs, and 0 inputs.

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

The analysis in [17] shows that 21 states and parameters are identifiable, assuming that all derivatives of the four outputs can be estimated. However, using Algorithm 4, we found that if no more than 26 derivatives are available, then only 9 states and parameters are identifiable. Further analysis is shown in Fig 3.

Variation of the number of identifiable parameters with maximum number of available output derivatives in a model of NF-κB regulatory module.The horizontal axis shows νmax: ∀i(j≤νmax↔yi(j) can be reliably estimated). The vertical axis shows the number of identifiable states and parameters in the model.

2.6 Limitations and questions for further research

The definitions and algorithms presented here naturally propose paths for continued research. Our approach requires the modeler to have already determined or assumed the maximum derivatives that can be estimated reliably. This raises the following questions:

Given a situation where the noise level is fixed, how can we determine the orders of the highest derivatives that can be reliably used to identify parameter values?Conversely, if one has decided what parameters and states must be identifiable for an experiment to be useful, how can one determine the maximum acceptable noise level so that the derivatives necessary for identification are known precisely enough?

3 Theory

The definition of $[eqn]$ -identifiability and the justification of our results are made rigorous in Sect 5. We give a brief summary of this theory here.

The notion of identifiability involves many subtleties. Our precise definition is a type of generic local single-experiment identifiability. This is done in terms of uniqueness (up to a finite set) of parameter and state values given known values of the output derivatives $[eqn]$ at a certain point in time. We show that if all derivatives of all outputs are available, then our definition is equivalent to the notion of local identifiability given in [15, Def. 2.5].

Our definition of $[eqn]$ -identifiability is intended to capture the property that a modeler really wants to ascertain, but it is often difficult to verify directly. Therefore we show that this property is equivalent to a property involving ranks of matrices, which is straightforward to check using existing software. This is in fact the property alluded to in Algorithm 1.

Although this rank property is straightforward to check, it involves computing ranks of symbolic matrices, which can cost the modeler time. Therefore, to alleviate this problem, we present probabilistic algorithms that substitute random integers for the variables and compute modulo a random prime (Algorithms 3 and 5.3.1). These are modifications of the method described in [29]. We present detailed probabilistic statements that a reader can use to create an implementation without investigating the proofs. We present detailed proofs that the reader can check without excessive pencil and paper computations.

Moreover, our work addresses an oversight in [29] surrounding the probability that a rational expression vanishes when evaluated at a random tuple of integers modulo a random prime. The author assumes that the denominator does not vanish, but then gives a bound for the probability that the numerator does not vanish regardless of this condition. We give a bound on the probability that the numerator does not vanish given that the denominator does not vanish.

4 Conclusion

Practical identifiability analysis lies at the heart of model-based data analysis. State-of-the-art numerical methods, such as profile likelihood and MCMC, suffer from the limitation that they require an exhaustive coverage of the parameter space, which is not possible for large and realistic models. This coverage is not needed for structural methods, but these methods traditionally assume that one can determine all derivatives of the measurements y. Such derivatives are not available in practice, but the consequences of this unfulfilled assumption have not been examined, because corresponding methods and algorithms are missing. Herein, we present a new definition, $[eqn]$ -identifability, which restricts the analysis to those derivatives of y that are available. We also present new algorithms that can determine such identifiability in practice. Applications to previously published models demonstrate that we can determine not only identifiability of parameters, but observability of any model property (Examples 2.5 and 2.6). These results allow us to strengthen previous conclusions. For instance, the previous rejection of a specific model for insulin signaling was based on the model-guided experiments measuring the amount of internalized and dephosphorylated insulin receptors (Example 2.5), and we now show that this model property indeed can be identifiable, using a methodology that does not require coverage of the entire parameter space. Our analysis also shows that the number of parameters identifiable according to traditional structural methods are widely overestimated. If one, e.g., assumes that only up to third order derivatives, instead of all derivatives, are available, the number of identifiable parameters drops from 17 to 1 for the Drosophila model (Fig 2), and from 21 to 4 for an NF- $[eqn]$ B model (Fig 3). In both models, the previously obtained identifiability is present only if at least 20 derivatives of all measurement signals are available. Our results bring us one step closer to a structural approach for practical identifiability analysis.

5 Appendix: Technical details

5.1 Notation and definitions

The notion of identifiability involves many subtleties. This subsection provides the setup necessary for rigorous treatment.

We repeat the setup described previously. Let $[eqn]$ be a non-negative integer and let n, m, and r be positive integers. Let $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ . Let $[eqn]$ and $[eqn]$ be tuples of rational functions in x, u, and $[eqn]$ over $[eqn]$ . This setup determines a class of systems of ODEs with initial conditions, or a model class:

$$[eqn]$$

Fix non-negative integer $[eqn]$ , positive integers n, m, and r, n-tuple f and m-tuple g of elements in $[eqn]$ for the rest of this section. The symbol $[eqn]$ denotes the set of non-negative integers.

Notation 5.1. (Differential Algebraic Setting)

(a) A differential ring $[eqn]$ is a ring together with a derivation, that is, a function $[eqn]$ satisfying $[eqn]$ and $[eqn]$ . All rings are assumed to be commutative and with multiplicative identity. For $[eqn]$ and $[eqn]$ , $[eqn]$ denotes the image of a after k applications of $[eqn]$ . The set $[eqn]$ will be called the set of derivatives of a.

(b) Let $[eqn]$ be a differential ring and let $[eqn]$ be a set of indeterminates. Then $[eqn]$ denotes the polynomial ring in infinitely many indeterminates $[eqn]$ with derivation $[eqn]$ extended by $[eqn]$ .

(c) Let $[eqn]$ be a differential subring of $[eqn]$ and let $[eqn]$ . Then $[eqn]$ denotes the smallest subring of S_2_ containing S_1_ and the derivatives of the elements of Z.

(d) Let $[eqn]$ be a differential ring that is a subring of a differential integral domain $[eqn]$ and let $[eqn]$ . Then $[eqn]$ denotes the fraction field of S_1_{Z}. This is a differential ring under the extension of $[eqn]$ via the quotient rule $[eqn]$ .

(e) For any differential ring containing any elements of $[eqn]$ we assume $[eqn]$ maps each such element to 0.

(f) Let $[eqn]$ be a differential ring. A differential ideal I is an ideal satisfying $[eqn]$ . For subset $[eqn]$ , we denote by [A] the smallest differential ideal containing A.

(g) For an ideal I and element a in a ring S, we denote $[eqn]$ . This set is also an ideal in S.

(h) Write $[eqn]$ and $[eqn]$ as fractions of elements of $[eqn]$ , let $[eqn]$ be the LCM of the denominators, and let $[eqn]$ and $[eqn]$ be such that $[eqn]$ and $[eqn]$ . Note that Q is unique up to multiplication by an element of $[eqn]$ and choice of Q will not affect our definitions and results. We define the differential ideal of $[eqn]$ as $[eqn]$ . This is a prime (differential) ideal (cf [15, Lemma 3.2]) and thus $[eqn]$ is a differential ring that is an integral domain.

(i) For $[eqn]$ , we denote by $[eqn]$ the subset $[eqn]$ of R. We define $[eqn]$ .

Notation 5.2.1 (Analytic Setting)

(a) Let $[eqn]$ . Then $[eqn]$ denotes the set of all functions that are complex analytic in some neighborhood of t = t_0_.

(b) Let $[eqn]$ . Then $[eqn]$ denotes the complement to the set where at least one of the denominators of f or g vanishes at t = t_0_.

(c) For $[eqn]$ and $[eqn]$ , let $[eqn]$ .

(d) For $[eqn]$ and $[eqn]$ , let $[eqn]$ and $[eqn]$ denote the unique solution of the initial value problem $[eqn]$ (cf. [1, Theorem 2.2.2]). For appropriate integers j and k, let $[eqn]$ denote the k-th derivative of the j-th component of $[eqn]$ .

(e) For $[eqn]$ , a subset $[eqn]$ is called Zariski open if there exists P in $[eqn]$ such that U is the complement to the zero set of P.

(f) For $[eqn]$ and $[eqn]$ , a subset $[eqn]$ is called Zariski open if there exists P in $[eqn]$ such that

$$[eqn]$$

(g) For $[eqn]$ , $[eqn]$ , and $[eqn]$ or $[eqn]$ , the set of all nonempty Zariski open subsets of W will be denoted by $[eqn]$ .

(h) A subset $[eqn]$ is called codiscrete if for any distinct $[eqn]$ there exist disjoint open sets $[eqn]$ and $[eqn]$ such that $[eqn]$ and $[eqn]$ .

Notation 5.3. Let $[eqn]$ , let $[eqn]$ , and let $[eqn]$ .

(a) Substituting $[eqn]$ for $[eqn]$ in h gives a function of one variable. The image of h under this substitution will be denoted by $[eqn]$ .

(b) The connected component containing t_1_ of the intersection of the domains of $[eqn]$ , $[eqn]$ , $[eqn]$ and $[eqn]$ will be denoted by $[eqn]$ .

Definition 5.4. Let $[eqn]$ . Let $[eqn]$ . The expression h is said to be $[eqn]$ -identifiable if

$$[eqn]$$ $$[eqn]$$

where

$$[eqn]$$

One can interpret this definition as follows: If at some time t0, not belonging to a certain discrete “singular” set, we know the exact values of $[eqn]$ , then if we consider all possible values of the state variables and parameters at t0 that produce these values of $[eqn]$ , and then we consider the set of values of h obtained by evaluating h at these values, this set is finite.

5.1.1 Examples to illustrate the definition.

Example 5.5. Consider the system

$$[eqn]$$

We show that $[eqn]$ is (1,0)-identifiable. Note that n = 1 and m = 2. Since two parameters occur we can take $[eqn]$ to be any integer at least 2. Although no inputs occur the definitions require that we take r to be at least 1. Set $[eqn]$ and r = 1. Let $[eqn]$ and let $[eqn]$ . Note that since there are no denominators we have $[eqn]$ for all t_0_. Fix $[eqn]$ . Let $[eqn]$ and fix $[eqn]$ . It is trivial to obtain a formula for the solution of this system, and we have

$$[eqn]$$

Let $[eqn]$ , and note that

$$[eqn]$$

For $[eqn]$ , the condition $[eqn]$ gives

$$[eqn]$$

By the definition of $[eqn]$ , it follows that no side of the first or third of these equations is 0. Dividing the second equation by the product of the first and third proves that $[eqn]$ . Since $[eqn]$ , we see that $[eqn]$ and thus has cardinality 1. We conclude that $[eqn]$ is (1,0)-identifiable.

We now show that $[eqn]$ is not (0,0)-identifiable. Fix $[eqn]$ and $[eqn]$ . Let $[eqn]$ . Fix codiscrete V, and let $[eqn]$ . Using the first and third equations of (4), we see that for all $[eqn]$ the tuple $[eqn]$ satisfies the condition $[eqn]$ for $[eqn]$ . Therefore

$$[eqn]$$

and the right-hand side is infinite. We conclude that $[eqn]$ is not (0,0)-identifiable.

The next example includes an input and demonstrates why sometimes a discrete set must be excluded from V.

Example 5.6. Consider the system

$$[eqn]$$ $$[eqn]$$

where f_2_ is some rational function.

We have n = 2 and m = 2. Take $[eqn]$ and r = 1. We show that x_2_ is (0,0)-identifiable. We have $[eqn]$ . Let $[eqn]$ and let $[eqn]$ . Let $[eqn]$ . Note that because of the way U was defined, $[eqn]$ is not the zero function. Let V be the complement of the vanishing of $[eqn]$ , which is necessarily codiscrete since $[eqn]$ and $[eqn]$ are analytic. Let $[eqn]$ . If $[eqn]$ is such that $[eqn]$ for $[eqn]$ then it follows that

$$[eqn]$$

Since $[eqn]$ we know that $[eqn]$ and $[eqn]$ are defined and since $[eqn]$ we know that $[eqn]$ . Thus $[eqn]$ and we conclude that x_2_ is (0,0)-identifiable.

5.1.2 ν-identifiable implies single-experiment locally identifiable and the converse is true for sufficiently large ν.

Many notions of identifiability are used in the literature. Some of these are stated and compared in [5]. In general, Definition 5.4 is not equivalent to any published definition as far as we are aware.

Definition 5.4 can be viewed as a type of single-experiment generic local identifiability. It is generic because there is a (possibly empty) set of numerical values of parameters and input functions for which h cannot be recovered but if the true values of the parameters and inputs lie outside this set h can be recovered. It is local because we allow h to lie in a finite set. One could change “is finite” to “equals one” to create a definition of a globally $[eqn]$ -identifiable function, however we do not address this in the current work. It is “single-experiment” because it refers to only one instance of the model. This is in contrast to a multi-experiment approach where one observes multiple instances of the model, usually with the same equation parameters but different initial conditions.

Since the equations of $[eqn]$ are rational in x, $[eqn]$ , and u, there exist relations among sufficiently high derivatives of y and hence it is not meaningful to consider $[eqn]$ beyond a certain order. This is made precise by the following proposition.

Proposition 5.7. Let $[eqn]$ and $[eqn]$ . For each $[eqn]$ let $[eqn]$ be the greatest non-negative integer such that the set $[eqn]$ is not algebraic over $[eqn]$ . Then h is $[eqn]$ -identifiable if and only if h is $[eqn]$ -identifiable. Moreover, $[eqn]$ .

Proof: Since the transcendence degree of $[eqn]$ over $[eqn]$ is $[eqn]$ , it must be that $[eqn]$ is algebraic over $[eqn]$ . Hence such a $[eqn]$ exists and moreover $[eqn]$ .

For the $[eqn]$ direction, note that if $[eqn]$ are such that $[eqn]$ for all i, it follows from the definition that h is b-identifiable implies h is a-identifiable.

We now address the $[eqn]$ direction. Suppose h is $[eqn]$ -identifiable. By Proposition 5.12, h is algebraic over $[eqn]$ . For any i, by writing an algebraic dependence of $[eqn]$ over $[eqn]$ and differentiating (noting that the field has characteristic 0), we see that for all $[eqn]$ the element $[eqn]$ is algebraic over $[eqn]$ . It follows that h is algebraic over $[eqn]$ . By Proposition 5.12 h is $[eqn]$ -identifiable. $[eqn]$

Definition 5.8. Let $[eqn]$ . The expression h is said to be single-experiment locally identifiable (SELI) if

$$[eqn]$$

where

$$[eqn]$$

For $[eqn]$ , Definition 5.8 is equivalent to the definition of local identifiability given in Definition 2.5 of [15] (generalization to multiple inputs is asserted in Remark 2.2). In [15, Prop. 3.4 (a) $[eqn]$ (c)], it was shown that for $[eqn]$ , h is SELI if and only if h is algebraic over $[eqn]$ . We extend this result to arbitrary $[eqn]$ .

Proposition 5.9. Let $[eqn]$ . Then h is SELI if and only if h is algebraic over $[eqn]$ .

Proof: Let $[eqn]$ be the system obtained by adding to $[eqn]$ the equations $[eqn]$ , $[eqn]$ , and $[eqn]$ . Note that $[eqn]$ in $[eqn]$ is the projection of $[eqn]$ onto all coordinates but xn + 1. We divide the proof into the following three steps: h is $[eqn]$ -SELI $[eqn]$ is $[eqn]$ -SELI $[eqn]$ is algebraic over $[eqn]$ h is algebraic over $[eqn]$ .

We show h is $[eqn]$ -SELI $[eqn]$ is $[eqn]$ -SELI. Suppose h is $[eqn]$ -SELI. Let $[eqn]$ and U be as required by the definition. Define $[eqn]$ . Let $[eqn]$ . We verify that $[eqn]$ . Since $[eqn]$ , we have

$$[eqn]$$

where $[eqn]$ . It follows that

$$[eqn]$$

where $[eqn]$ is considered a subset of $[eqn]$ . Thus

$$[eqn]$$

and we know that $[eqn]$ because h is $[eqn]$ -SELI. This completes the first direction. Now suppose xn + 1 is $[eqn]$ -SELI. Let $[eqn]$ and U be as required by the definition. Define $[eqn]$ to be the projection of $[eqn]$ onto all coordinates other than xn + 1. Let $[eqn]$ . Let $[eqn]$ be such that $[eqn]$ . Such a $[eqn]$ exists because of the way $[eqn]$ was defined. Now

$$[eqn]$$

Hence

$$[eqn]$$

and we know that $[eqn]$ because xn + 1 is $[eqn]$ -SELI.

From [15, Prop. 3.4 (a) $[eqn]$ (c)], we have that xn + 1 is $[eqn]$ -SELI $[eqn]$ is algebraic over $[eqn]$ .

We now show that h is algebraic over $[eqn]$ is algebraic over $[eqn]$ . Suppose h is algebraic over $[eqn]$ . Then $[eqn]$ is algebraic over $[eqn]$ . We now address the other direction. Let $[eqn]$ and let $[eqn]$ . Suppose xn + 1 is algebraic over $[eqn]$ . Then $[eqn]$ is algebraic over $[eqn]$ . Let $[eqn]$ be the minimal polynomial of h over $[eqn]$ . Suppose ym + 1 appears in some ai. Let $[eqn]$ be the differential field automorphism on $[eqn]$ such that $[eqn]$ is fixed pointwise, $[eqn]$ , and $[eqn]$ . Now $[eqn]$ is a non-zero polynomial of lower degree with coefficients in $[eqn]$ that has h as a root. Thus we have a contradiction. Therefore all the ai lie in $[eqn]$ and h is algebraic over $[eqn]$ . $[eqn]$

This leads to the main result of this section.

Corollary 5.10. Let $[eqn]$ and $[eqn]$ .

h is $[eqn]$ -identifiable $[eqn]$ h is SELI.

If $[eqn]$ , then h is SELI $[eqn]$ h is $[eqn]$ -identifiable.

Proof: Suppose h is $[eqn]$ -identifiable. Then by Proposition 5.12 we know h is algebraic over $[eqn]$ . It follows that h is algebraic over $[eqn]$ and then by Proposition 5.9 we have that h is SELI.

Suppose that $[eqn]$ . Then by the arguments presented in Proposition 5.7 we have $[eqn]$ . Suppose h is SELI. By Proposition 5.9 we have that h is algebraic over $[eqn]$ , which equals $[eqn]$ . By Proposition 5.12 we have that h is $[eqn]$ -identifiable. $[eqn]$

5.2 Proof of algorithms

The definition of $[eqn]$ -identifiability is stated in terms of analytic functions. The Proposition 5.12 gives a correspondence between the analytic property and an algebraic property. Its proof will use the following lemma.

Lemma 5.11. Let $[eqn]$ , and let $[eqn]$ . The map

$$[eqn]$$

gives a $[eqn]$ -algebra homomorphism $[eqn]$ , where S is the ring $[eqn]$ . Moreover, under this map $[eqn]$ .

Proof: Denote the stated map by $[eqn]$ . Since $[eqn]$ satisfy all the relations among $[eqn]$ and no denominator of S is sent to 0, we see that $[eqn]$ is a homomorphism on S. Now $[eqn]$ , so $[eqn]$ . The existence and uniqueness theorem guarantees the existence and uniqueness of $[eqn]$ , and $[eqn]$ is given by $[eqn]$ . $[eqn]$

Proposition 5.12. Let $[eqn]$ . Expression $[eqn]$ is $[eqn]$ -identifiable if and only if h is algebraic over the subfield $[eqn]$ of Frac(R).

Proof: Part 1: $[eqn]$ Assume that h is algebraic over $[eqn]$ . Consider the minimal polynomial of h over $[eqn]$ . Clearing denominators, we obtain the polynomial $[eqn]$ , where each $[eqn]$ and $[eqn]$ . By [16, Corollary 6.6], there exist $[eqn]$ and $[eqn]$ such that for all $[eqn]$ it holds that $[eqn]$ (see Notation 5.3) is not the zero function. Fix such $[eqn]$ and U and choose a $[eqn]$ . Let V be the complement of the vanishing of $[eqn]$ . Since $[eqn]$ is analytic about 0, we know that V is codiscrete.

Fix $[eqn]$ . Let $[eqn]$ , $[eqn]$ be such that $[eqn]$ and $[eqn]$ . Since each ai belongs to $[eqn]$ , it follows from Lemma 5.11 that $[eqn]$ for all i. Applying Lemma 5.11 to the equation $[eqn]$ , we find that $[eqn]$ . Thus $[eqn]$ is a root of the non-zero polynomial $[eqn]$ . Noting that $[eqn]$ we conclude that $[eqn]$ is finite.

Part 2: $[eqn]$ Assume that h is not algebraic over $[eqn]$ and that h is $[eqn]$ -identifiable. We will give a proof by contradiction. The proof can be divided into four steps:

Step 0 Label the rings that will be used in the proof.

Step 1 Choose $[eqn]$ . This will be done in terms of the non-vanishing of minimal polynomials over fraction fields of intermediate rings. Fix V and t0 and note that Lemma 5.11 with $[eqn]$ gives a ring homomorphism $[eqn]$ .

Step 2 Show that the set $[eqn]$ is infinite. This will involve careful extension of ring homomorphisms.

Step 3 Verify that $[eqn]$ is infinite by noting that each $[eqn]$ corresponds to a tuple $[eqn]$ .

We begin the proof.

Step 0 Label $[eqn]$ as $[eqn]$ . Without loss of generality, let k be such that $[eqn]$ if and only if bj is algebraic over $[eqn]$ . Let R2 equal $[eqn]$ and let F2 be its field of fractions. Let $[eqn]$ be such that $[eqn]$ is a transcendence basis for $[eqn]$ over F2, relabeling if necessary. Let $[eqn]$ and let F3 be its field of fractions. Now $[eqn]$ are each algebraic over $[eqn]$ . For $[eqn]$ let Pj(Z) be the minimal polynomial of bj over F4 multiplied by the LCM of the denominators, where F4 is viewed as the field of fractions of $[eqn]$ . For each $[eqn]$ , let $[eqn]$ be the leading coefficient of Pj. Write $[eqn]$ , where $[eqn]$ . Let $[eqn]$ be the minimal polynomial of $[eqn]$ over F4 multiplied by the LCM of the denominators. (Note that if $[eqn]$ then $[eqn]$ .) Let $[eqn]$ and $[eqn]$ be its leading and constant coefficients, respectively.

Table 1: Intermediate rings and their fraction fields.

Step 1 Let $[eqn]$ and U be as guaranteed by the definition of $[eqn]$ -identifiability. Consider the subset of $[eqn]$ consisting of all $[eqn]$ such that for all $[eqn]$ $[eqn]$ the expressions Lj, $[eqn]$ , and $[eqn]$ evaluated at $[eqn]$ and then t = 0 are not zero. This subset is non-empty because it is the intersection of finitely many non-empty Zariski open sets. Hence its intersection with $[eqn]$ is non-empty. Fix such a $[eqn]$ . Let V be as required by the definition of $[eqn]$ -identifiability, let $[eqn]$ (recall Notation 5.3), and let $[eqn]$ be the $[eqn]$ -algebra homomorphism from R[h] to $[eqn]$ given by applying the ring homomorphism described in Lemma 5.11 with $[eqn]$ .

Step 2 We now show that there are infinitely many $[eqn]$ such that there exists a $[eqn]$ -algebra homomorphism $[eqn]$ such that $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Define $[eqn]$ on R2 by $[eqn]$ . For each j, define $[eqn]$ to be the element obtained by evaluating the coefficients (in R2) of Lj via $[eqn]$ ; define the analogous expressions with Lj replaced by $[eqn]$ and $[eqn]$ . By the way $[eqn]$ was chosen, we have that $[eqn]$ , $[eqn]$ , and all $[eqn]$ are non-zero. Using Lemma 5.13 we can extend $[eqn]$ to $[eqn]$ in a way that makes neither $[eqn]$ nor $[eqn]$ nor any $[eqn]$ equal to zero. Choose such an extension and call this, somewhat abusing notation, $[eqn]$ . Since h is not algebraic over F3, by Lemma 5.13 $[eqn]$ can be extended by mapping h to any element of $[eqn]$ . Now there are infinitely many c such that $[eqn]$ . Fix such a c for the remainder of the proof and extend $[eqn]$ to R4 by letting $[eqn]$ .

We now extend $[eqn]$ to $[eqn]$ . By the preceding discussion, the leading and constant coefficients of $[eqn]$ are non-zero, so by Lemma 5.13 we can extend $[eqn]$ to $[eqn]$ so that $[eqn]$ .

Next we extend $[eqn]$ to $[eqn]$ . Observe that the minimal polynomial $[eqn]$ of $[eqn]$ over $[eqn]$ is a factor of $[eqn]$ . Therefore $[eqn]$ is a non-zero factor of $[eqn]$ . By Lemma 5.13 we can extend $[eqn]$ to $[eqn]$ .

The argument from the preceding paragraph can be repeated to show that $[eqn]$ can be extended to $[eqn]$ . Applying this several more times, we see that we can extend $[eqn]$ to R[h].

Let $[eqn]$ . We have shown that $[eqn]$ is infinite.

Step 3 We will now show that $[eqn]$ is infinite. We have

$$[eqn]$$

By Lemma 5.11 it holds that $[eqn]$ and $[eqn]$ . Hence

$$[eqn]$$

Since for each $[eqn]$ we have $[eqn]$ , the first conjunct is satisfied. By the definition of T, the second conjunct is satisfied for all $[eqn]$ . Hence

$$[eqn]$$

We showed in Step 2 that the right hand side is infinite, and thus $[eqn]$ is infinite. Therefore h is not $[eqn]$ -identifiable, contradicting our assumption. $[eqn]$

Lemma 5.13. Let W be a (possibly infinite) set of indeterminates and let S be a $[eqn]$ -subalgebra of the field of rational functions $[eqn]$ . Let $[eqn]$ be a $[eqn]$ -algebra homomorphism. Let $[eqn]$ .

Suppose f is not algebraic over $[eqn]$ . Then $[eqn]$ can be extended to S[f] by mapping f to any element of $[eqn]$ .

Suppose f is algebraic over $[eqn]$ . Let P(Z) be the minimal polynomial of f over $[eqn]$ multiplied by the LCM of the denominators. Write $[eqn]$ , where $[eqn]$ . If $[eqn]$ , then $[eqn]$ can be extended to S[f]. If furthermore $[eqn]$ , then in this extension $[eqn]$ .

Proof: Suppose f is not algebraic over $[eqn]$ . Our result follows from [2, p. 99].

Suppose f is algebraic over $[eqn]$ and $[eqn]$ . If $[eqn]$ then the result is trivial. Assume f is not in $[eqn]$ . By [2, Theorem 3.2 p. 347], $[eqn]$ can be extended to S[f] or $[eqn]$ . If the former is true the proof is complete. Suppose $[eqn]$ can be extended to $[eqn]$ . Writing $[eqn]$ , we have that $[eqn]$ . If $[eqn]$ , then it follows that $[eqn]$ , which contradicts our hypotheses. Thus $[eqn]$ . Now we can extend $[eqn]$ to the ring $[eqn]$ , which is equal to the subring of $[eqn]$ consisting of fractions with numerators in $[eqn]$ and denominators in $[eqn]$ (cf. [2, p. 346]). Since $[eqn]$ , the element f belongs to $[eqn]$ . By restricting $[eqn]$ to S[f] we have an extension of $[eqn]$ to S[f].

Still assuming f is algebraic over $[eqn]$ , suppose $[eqn]$ . The image of f must satisfy $[eqn]$ . Thus it is impossible that $[eqn]$ . ◻

It is not always obvious whether a given field element is algebraic over a given subfield. The following proposition gives an equivalence that, for our purposes, reduces this question to the problem of checking the rank of a matrix with easily computable entries. We will use the following notation:

Notation 5.14. Let $[eqn]$ be elements of a $[eqn]$ -algebra and let $[eqn]$ be an algebraically independent set over $[eqn]$ . We denote by $[eqn]$ the matrix whose (i,j)-th entry is $[eqn]$ . If M is such a matrix and $[eqn]$ , then $[eqn]$ denotes the result when the column corresponding to Z_0_ is removed. If this column does not appear in M then $[eqn]$ is equal to M.

We will use the following algebraic fact, which generalizes [12, Thm. 2.3].

Proposition 5.15. Let $[eqn]$ each be algebraic over $[eqn]$ , where $[eqn]$ and all w_i_ and z_i_ are indeterminates. The elements $[eqn]$ are algebraically independent over $[eqn]$ if and only if the matrix $[eqn]$ has rank equal to s.

Proof: Let $[eqn]$ and let $[eqn]$ $[eqn]$ . Note that

$$[eqn]$$

where I is the $[eqn]$ identity matrix and A is an $[eqn]$ matrix.

Now $[eqn]$ is algebraically independent over $[eqn]$ iff $[eqn]$ is algebraically independent over $[eqn]$ iff (by [12, Thm. 2.3]) $[eqn]$ iff $[eqn]$ . $[eqn]$

Proposition 5.16. Let $[eqn]$ and let $[eqn]$ . Then h is algebraic over $[eqn]$ if and only if the matrix $[eqn]$ has the same rank as $[eqn]$ .

Proof: Note that a subset of $[eqn]$ is algebraically independent over $[eqn]$ if and only if it is algebraically independent over $[eqn]$ . Let T be a maximal subset of $[eqn]$ that is algebraically independent over $[eqn]$ . Then by Proposition 5.15 with $[eqn]$ and $[eqn]$ we have $[eqn]$ .

Suppose h is algebraic over $[eqn]$ . Then $[eqn]$ is algebraic over $[eqn]$ . Then $[eqn]$ by Proposition 5.15. Therefore $[eqn]$ .

Suppose h is not algebraic over $[eqn]$ . Then $[eqn]$ is not algebraic over $[eqn]$ . Then $[eqn]$ by Proposition 5.15. $[eqn]$

Corollary 5.17. Algorithm 1 always terminates. The output is “Yes” if and only if h is $[eqn]$ -identifiable.

Proof: Termination is obvious. The other result follows from Proposition 5.12 and Proposition 5.16. $[eqn]$

Proposition 5.18. Let $[eqn]$ and let J be as in Algorithm 2. Let $[eqn]$ . Then z is algebraic over $[eqn]$ if and only if $[eqn]$ .

Proof: Without loss of generality we assume $[eqn]$ . By Proposition 5.16, $[eqn]$ is algebraic over $[eqn]$ if and only if $[eqn]$ . Now

$$[eqn]$$

$[eqn]$

Corollary 5.19. Algorithm 2 always terminates. The output is “Yes” if and only if $[eqn]$ is $[eqn]$ -identifiable.

Proof: Termination is obvious. The other result follows from Proposition 5.12 and Proposition 5.18. $[eqn]$

5.3 Probabilistic method for improved speed

5.3.1 Presentation of algorithms and summary of results.

Algorithm 1 and 2 involve computing the rank of a matrix of rational expressions. It is usually much faster to insert random numbers for the variables and compute the rank of the resulting matrix. The disadvantage of this is that the numerical matrix may have lower rank than the symbolic. In [28] a method for doing this in the case where the coefficients of $[eqn]$ are integers with user-specified probability of success was given. We adapt that method to our algorithms.

For the rest of this section, assume the coefficients of f and g in (2) and h in Algorithm 1 are integers. Note that the entries of J belong to $[eqn]$ . Our strategy involves randomly choosing $[eqn]$ non-negative integers and a prime number, and then evaluating the determinant of our matrix at these integers modulo the prime. Algorithm 3 implements this strategy on Algorithm 1.

Algorithm 3 Determines whether parameter combination h is $[eqn]$ -identifiable.

Input : Equations $[eqn]$ , where $[eqn]$

$[eqn]$

$[eqn]$

$[eqn]$

Output : “Yes” if h is $[eqn]$ -identifiable with probability at least $[eqn]$

“No” if h is not $[eqn]$ -identifiable with probability at least $[eqn]$

Step 1a: Compute the least $[eqn]$ such that $[eqn]$ .

Step 1b: Compute D and the least N as in Proposition 5.25 with $[eqn]$ .

Step 1c: Choose $[eqn]$ uniformly at random.

Step 1d: If Q(T) ≡ 0 mod p, repeat Step 1c. Otherwise continue to Step 1e.

Step 1e: Compute Jh(T) mod p.

Step 2: Compute $[eqn]$ and rank (J(T) mod p).

Step 3: If $[eqn]$ , output “Yes”. Otherwise output “No”.

Algorithm 4 Determines the $[eqn]$ -identifiable subset of $[eqn]$ .

Input : Equations $[eqn]$ , where $[eqn]$

$[eqn]$

$[eqn]$

Output : Subset of $[eqn]$ consisting of exactly the $[eqn]$ -identifiable elements, with

probability at least $[eqn]$

Step 1a: Compute the least $[eqn]$ such that $[eqn]$ .

Step 1b: Compute D and the least N as in Proposition 5.25 with $[eqn]$ .

Step 1c: Choose $[eqn]$ uniformly at random.

Step 1d: If $[eqn]$ , repeat Step 1c. Otherwise continue to Step 1e.

Step 1e: Compute J(T) mod p.

Step 2: Compute $[eqn]$ .

Step 3: Let $[eqn]$ .

For $[eqn]$

Compute $[eqn]$ .

If $[eqn]$ , add z to Sout.

End For

Step 4: Return Sout.

While Algorithm 2 determines whether an individual parameter is $[eqn]$ -identifiable, Algorithm 4 uses this concept to determine, with user-specified probability, all elements of $[eqn]$ that are $[eqn]$ -identifiable.

The main results on Algorithm 3 and Algorithm 4 are the following:

The expected time to reach Step 1e is negligible. Once Step 1e is reached the algorithm is guaranteed to terminate. (Proposition 5.29)The probability that the output is correct is at least $[eqn]$ . (Propositions 5.30 and 5.31)For Algorithm 3, if in Step 2 $[eqn]$ , then h is $[eqn]$ -identifiable. For Algorithm 4, if in Step 2 $[eqn]$ , then every element of $[eqn]$ is $[eqn]$ -identifiable. (Proposition 5.32)

Remark 5.20. At the beginning of this section, we assumed n, m, and r are positive integers. As noted in Example 5.5, $[eqn]$ and r must be at least equal to the number of parameters and outputs, respectively, that appear in $[eqn]$ , but can be chosen to be greater without changing identifiability results. In Algorithms 5.3.1 and 5.3.1 it is sufficient to use $[eqn]$ and r equal to the number of parameters and inputs, respectively, appearing in the equations. In particular one can use r = 0 if no inputs appear and the main results on the algorithms are correct.

Remark 5.21. We have presented Algorithms 5.3.1 and 5.3.1 only for the case where f and g are not all elements of $[eqn]$ , since removing this restriction would require addressing special cases in several of the proofs. If $[eqn]$ , then h is $[eqn]$ -identifiable if and only if $[eqn]$ . One could easily add a step at the beginning of either algorithm to accommodate this case.

Remark 5.22. Our algorithms do not specify the methods used to compute the ranks of matrices. One can use the state-of-the-art method for this to achieve maximum speed.

5.3.2 Proof of algorithms.

In Algorithms 5.3.1 and 5.3.1, a random tuple of integers T and a random prime p are chosen, we check that the denominator of a determinant evaluated at T does not vanish modulo p, and then calculate the rank of a matrix after evaluating at T modulo p. We show that this gives the correct results with user-specified probability. The main results on Algorithms 3 and 5.3.1 are Propositions 5.29, 5.30, 5.31, and 5.32. The theory is based on bounds on integer roots of polynomials with integer coefficients, as well as the distribution of the prime numbers.

First, we use Proposition 5.23, Lemma 5.24, and Lemma 5.26 to prove Proposition 5.25, which gives conditions on the sets from which we choose T and p so that the numerator of the determinant evaluated at T does not vanish modulo p with user-specified probability. This proposition is essentially a more precisely stated version of [28, Proposition 6], and the proof we give is outlined in [28].

Next, Proposition 5.28 shows that the probability that the numerator vanishes given that the denominator does not vanish can be specified by the user. Note that this is the true probability associated with the algorithm, since we must first check that our choice of (T,p) does not make the denominator vanish before proceeding. This issue is not addressed in similar algorithms (cf. [29, p. 739], [17]) and is non-trivial, as shown by Example 5.27.

Finally, we prove statements about the algorithms that are directly relevant to helping the user interpret them. Although in principle, arbitrarily many instances of (T,p) may need to be chosen before finding one that does not make the denominator vanish, Proposition 5.29 shows that the expected time for a successful choice is negligible. It also asserts the algorithm’s termination after such a successful choice. Propositions 5.30 and 5.31 show that the algorithms produce the correct result with user-specified probability. Proposition 5.32 states that when the rank of the specialized matrix is full, the algorithms output the correct result with certainty.

Proposition 5.23 ([3] Prop. 98 p. 192). Let $[eqn]$ be a polynomial of total degree D over an integral domain A. Let $[eqn]$ . If an element $[eqn]$ is chosen from $[eqn]$ uniformly at random, then

$$[eqn]$$

Lemma 5.24 ([34] Lemma 18.9 p. 525). Let $[eqn]$ be a nonempty finite set of prime numbers, let $[eqn]$ , and let $[eqn]$ . If p is chosen from S uniformly at random, then

$$[eqn]$$

Proposition 5.25. Let J (resp. J_h_) be as in Algorithm 2 (resp. Algorithm 1) and suppose J (resp. J_h_) has at least one non-zero entry. Let

$[eqn]$

d _ 1 _ * =  maximum degree of the numerators and denominators of $[eqn]$ and $[eqn]$ (resp. $[eqn]$ , $[eqn]$ and h)*

$[eqn]$ $[eqn]$ , *  where $[eqn]$ (resp. $[eqn]$ )* C be such that $[eqn]$ , where $[eqn]$ (resp. $[eqn]$ ) $[eqn]$ *such that $[eqn]$ * $[eqn]$

Let J_0_ be a square submatrix of J (resp. J_h_) with rank equal to that of J (resp. J_h_). Note that the numerator of $[eqn]$ lies in $[eqn]$ . If a tuple of values T of the variables is chosen uniformly at random from $[eqn]$ and p is chosen uniformly at random from S, then

$$[eqn]$$

where $[eqn]$ represents the specialization of the numerator of $[eqn]$ at the chosen values.

Proof: We prove the main version first. The version for Jh will follow quickly from this.

Note that since J has at least one non-zero entry, d1 is well-defined and positive, and hence D is positive. Let W denote $[eqn]$ . We have that

$$[eqn]$$

By Proposition 5.23, we have that

$$[eqn]$$

We show that the degree of the numerator of any entry of J is no greater than $[eqn]$ . Fix $[eqn]$ . We first show by induction that for $[eqn]$ we can write $[eqn]$ , where $[eqn]$ . For the base case k = 0, recall from Notation 5.1 we have that $[eqn]$ , and $[eqn]$ . For the inductive hypothesis, fix $[eqn]$ and assume $[eqn]$ with $[eqn]$ . Applying the quotient rule we have that

$$[eqn]$$

recalling from Notation 5.1 that each $[eqn]$ . Since $[eqn]$ and $[eqn]$ do not exceed (n + m)d1, we conclude the inductive step. Continuing the proof of the bound on the degrees of the numerators of the entries of J, note that for any $[eqn]$ we can write $[eqn]$ where $[eqn]$ . So we can write $[eqn]$ where $[eqn]$ . In J0 the value of k does not exceed $[eqn]$ so we conclude the argument.

Since J0 is square it has at most $[eqn]$ rows. Therefore $[eqn]$ is no greater than D. Thus, we have

$$[eqn]$$

Assume that $[eqn]$ . Lemma 5.26 below shows that $[eqn]$ . Our assumptions imply that $[eqn]$ and hence $[eqn]$ . From [34, Exercise 18.18] it follows that S is non-empty. Using Lemma 5.24 with M = W, we have

$$[eqn]$$

By [34, Exercise 18.18], we have $[eqn]$ . It follows that

$$[eqn]$$

Combining (5), (7), and (8), we have our result.

We now prove the version with Jh. Let $[eqn]$ be the system obtained by adding the equation ym + 1 = h to $[eqn]$ and set $[eqn]$ . Now Jh for $[eqn]$ is equal to J for $[eqn]$ . Our result follows from the main version of the proposition. $[eqn]$

Lemma 5.26. In the setup of Proposition 5.25, $[eqn]$ .

Proof: For a polynomial p with integer coefficients, we shall define the height of p as $[eqn]$ . We will use the following properties ([28, Lemma 1]): For polynomials $[eqn]$ in $[eqn]$ variables, tuple of integers T0, and partial derivative $[eqn]$ , the following hold:

$[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Fix $[eqn]$ and write $[eqn]$ , we will prove by induction that for all $[eqn]$

$$[eqn]$$

We noted earlier that $[eqn]$ , all $[eqn]$ , and $[eqn]$ are no greater than (n + m)d1. It follows from this and the product height property that $[eqn]$ . For conciseness, we will use $[eqn]$ and $[eqn]$ to denote these degree and height bounds, respectively.

For the base case k = 0, note that $[eqn]$ . For the inductive hypothesis, fix $[eqn]$ and suppose Ak satisfies (9). As shown in the (6), we have

$$[eqn]$$

Using the derivation and product height properties, as well as the bound $[eqn]$ , we have

$$[eqn]$$ $$[eqn]$$

By the sum height property we have that

$$[eqn]$$

By the product height property we have

$$[eqn]$$

By the sum height property we have

$$[eqn]$$

Noting that $[eqn]$ , we conclude the inductive step.

Now for $[eqn]$ , we have that $[eqn]$ is equal to $[eqn]$ . A bound on the height of the numerator is given by the right-hand side of (10). Since RHS of (10) $[eqn]$ RHS of (11) $[eqn]$ RHS of (12), we see that a bound on this height is also given by the RHS of (9) with k replaced by k + 1. Observing that the maximum value of k occurring for an entry in J0 is $[eqn]$ and applying the evaluation height property, we have that the numerator of each entry of J0(T) is bounded by

$$[eqn]$$

Since our bounds assume that entries in the same row have the same denominator, we have that $[eqn]$ , where $[eqn]$ is the matrix whose elements are the numerators of J0. Henceforth we assume J0 has polynomial entries with heights bounded by B when evaluated at T. Using Hadamard’s Theorem, we have that $[eqn]$ , where $[eqn]$ is the square root of the sum of the squares of the elements of the i-th column of J0(T). Hence $[eqn]$ . Thus

$$[eqn]$$

Recalling that

$$[eqn]$$

we conclude the proof. $[eqn]$

After a set of random integers and a random prime number is chosen, we must first check that no denominator vanishes modulo the prime number before evaluating the rank of the matrix. Thus the probability that the algorithm gives the correct answer is not simply the probability that p does not divide $[eqn]$ , but rather the probability that p does not divide $[eqn]$ given that p does not divide $[eqn]$ . If a random tuple T is chosen and used to evaluate two polynomials A and B, it is not necessarily the case that $[eqn]$ , as the following example shows:

Example 5.27. Let k be a positive integer and let $[eqn]$ and $[eqn]$ . If an integer T is chosen uniformly at random from $[eqn]$ , then $[eqn]$ and $[eqn]$ .

The following proposition shows that the conditions used for Proposition 5.25 also give a bound on the conditional probability that is relevant to our algorithms.

Proposition 5.28. Let J (resp. J_h_) be as in Algorithm 2 (resp. Algorithm 1) and suppose J (resp. J_h_) has at least one non-zero entry. Let $[eqn]$ , D, S, and J_0_ be as in Proposition 5.25. If a tuple of values T of the variables is chosen uniformly at random from $[eqn]$ and p is chosen uniformly at random from S, then

$$[eqn]$$

where $[eqn]$ represents the specialization of the numerator of $[eqn]$ at the chosen values.

Proof: Let M denote the sample space $[eqn]$ . Let

$$[eqn]$$

Now

$$[eqn]$$

By Proposition 5.25 we have $[eqn]$ . We now prove the same bound holds for $[eqn]$ . Now $[eqn]$ , so using Proposition 5.23 we have $[eqn]$ . By the first statement of [28, Prop. 3] with j = 0 we have that the height of Q(T) is no greater than $[eqn]$ , which is no greater than $[eqn]$ . Thus $[eqn]$ and if $[eqn]$ then by Lemma 5.24 we have $[eqn]$ , which by [34, Exercise 18.18] is no greater than $[eqn]$ . Thus $[eqn]$ . Now we have

$$[eqn]$$

$[eqn]$

Proposition 5.29. Fix the input to Algorithm 3 (or Algorithm 4). (i) The probability that Step 1e will be reached in no more than k iterations of Step 1c is at least $[eqn]$ . (ii) When Step 1e is reached the algorithm is guaranteed to terminate.

Proof: (i) It was shown in the proof of Proposition 5.28 that $[eqn]$ , and based on the way $[eqn]$ was chosen in Step 1a it is trivial to verify that this is no greater than $[eqn]$ . Therefore for k independent choices the probability that $[eqn]$ for at least one of them is at least $[eqn]$ .

(ii) This is obvious. $[eqn]$

Proposition 5.30. Fix the input to Algorithm 3. If h is $[eqn]$ -identifiable, the probability that the output is “Yes” is at least $[eqn]$ . If h is not $[eqn]$ -identifiable, the probability that the output is “No” is at least $[eqn]$ .

Proof: Consider the following subsets of $[eqn]$ : $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ . We have

$$[eqn]$$

Proposition 5.28 gives us that $[eqn]$ and $[eqn]$ .

Suppose h is $[eqn]$ -identifiable. By Propositions 5.12 and 5.16, we have $[eqn]$ . We show that $[eqn]$ . Suppose $[eqn]$ . Then $[eqn]$ , and we deduce that $[eqn]$ , so the algorithm outputs “Yes”.

Suppose h is not $[eqn]$ -identifiable. By Propositions 5.12 and 5.16, we have $[eqn]$ . We show that $[eqn]$ . Suppose $[eqn]$ . Then $[eqn]$ , so the algorithm outputs “No”. $[eqn]$

Proposition 5.31. *Fix the input to Algorithm 4. Let S_id_ denote the $[eqn]$ -identifiable subset of $[eqn]$ and let S_out_ denote the output of Algorithm 4. The probability that $[eqn]$ is at least $[eqn]$ *.

Proof: Because Sout depends on (T,p) we shall use the notation Sout(T,p). Consider the following subsets of $[eqn]$ : $[eqn]$ and $[eqn]$ . We have

$$[eqn]$$

By Proposition 5.28 we have $[eqn]$ . We show $[eqn]$ . Suppose $[eqn]$ . Let $[eqn]$ . Suppose z is $[eqn]$ -identifiable. By Propositions 5.12 and 5.16, we have $[eqn]$ . Now $[eqn]$ . Since the final row of Jz(T) must be $[eqn]$ , it follows that $[eqn]$ . Therefore $[eqn]$ . Suppose z is not $[eqn]$ -identifiable. It follows from Propositions 5.12 and 5.16 that $[eqn]$ . Because $[eqn]$ and the final row of Jz(T) is $[eqn]$ we have that $[eqn]$ . Now $[eqn]$ . Therefore $[eqn]$ . $[eqn]$

Proposition 5.32. Suppose that in Step 2 of Algorithm 3 (resp. Algorithm 4) $[eqn]$ . Then h is $[eqn]$ -identifiable and the output is “Yes” (resp. every element of $[eqn]$ is $[eqn]$ -identifiable and the output is $[eqn]$ ).

Proof: We first prove the statement regarding Algorithm 3. We have $[eqn]$ , so $[eqn]$ . It follows that $[eqn]$ and by Propositions 5.12 and 5.16 we have that h is $[eqn]$ -identifiable. Since $[eqn]$ , it follows that $[eqn]$ and the output will be “Yes”.

We now address Algorithm 4. By the preceding paragraph $[eqn]$ . Since for any $[eqn]$ the matrix $[eqn]$ has only $[eqn]$ columns, it must be that $[eqn]$ and hence by Propositions 5.12 and 5.18 each element of $[eqn]$ is $[eqn]$ -identifiable. Similarly, $[eqn]$ and hence the output is $[eqn]$ . $[eqn]$

Supporting information

S1 FileMaple code for algorithm 4.Currently contains entries for Drosophila period protein model.(PDF)

S2 FileMaple code for algorithm 3.Currently contains entries for NF- $[eqn]$ B model.(MW)

S3 TableDescription of models.Details of the models used in the paper.(MW)