A comparison of evaluation methods in coevolution

TL;DR

This paper compares four evaluation methods in coevolution using the Majority Function problem, finding that combining average score and weighted informativeness yields more accurate assessments.

Contribution

It introduces a comparison framework for evaluation methods in coevolution and demonstrates the effectiveness of combining specific measures for better evaluation accuracy.

Findings

Weighted informativeness improves evaluation accuracy

Combination of average score and weighted informativeness is more reliable

OF C is computed on a real problem for the first time

Abstract

In this research, we compare four different evaluation methods in coevolution on the Majority Function problem. The size of the problem is selected such that evaluation against all possible test cases is feasible. Two measures are used for the comparisons, i.e., the objective fitness derived from evaluating solutions against all test cases, and the objective fitness correlation (OFC), which is defined as the correlation coefficient between subjective and objective fitness. The results of our experiments suggest that a combination of average score and weighted informativeness may provide a more accurate evaluation in coevolution. In order to confirm this difference, a series of t-tests on the preference between each pair of the evaluation methods is performed. The resulting significance is affirmative, and the tests for two quality measures show similar preference on four evaluation methods. %This study is the first time OFC is actually computed on a real problem. Experiments on Majority Function problems with larger sizes and Parity problems are in progress, and their results will be added in the final version.

Tables

Table 1. Table 1: Design of experiments.

Exp No	1	2	3	4	5
Algorithm	GA	CO	CO	CO	CO
Evaluation Method	AS	AS	WS	AI	WI
Number of Runs	10	10	10	10	10

Table 2. Table 2: Parameters used for the majority function problem.

Parameter	Value
MAX_GENERATION	200
Size of population	64/100
Elitism	12/20 (20%)
Selection	linear rank selection
Crossover	one point crossover
Mutation rate	0.01

Table 3. Table 3: Significant level in paired t-tests for objective fitness among experiment COAS, COWS, COAI and COWI. A p-value smaller than 0.05 is usually considered as significant.

P-value for test
on i j	WS	AI	WI
AS	$1.4\times {10}^{-21}$	$9.9\times {10}^{-15}$	$1.9\times {10}^{-68}$
WS		$9.8\times {10}^{-1}$	$7.2\times {10}^{-81}$
AI			$1.5\times {10}^{-64}$

Table 4. Table 4: Significant level in paired t-tests for OFC among experiment COAS, COWS, COAI and COWI.

P-value for test
on i j	WS	AI	WI
AS	$4.2\times {10}^{-5}$	$6.5\times {10}^{-21}$	$2.9\times {10}^{-69}$
WS		$4.6\times {10}^{-9}$	$3.4\times {10}^{-59}$
AI			$1.1\times {10}^{-39}$

Equations

$$dist(T_i,S_k,S_l)\iff G(T_i,S_k)>G(T_i,S_l)$$

$$F_{AS}(T_i)$$

$$F_{AS}(S_j)$$

$$W_T(T_i)$$

$$W_S(S_j)$$

$$W_T^{\prime }(T_i)$$

$$W_S^{\prime }(S_j)$$

$$F_{WS}(T_i)$$

$$F_{WS}(S_j)$$

$$D_T(T_i)$$

$$D_S(S_j)$$

$$D_T^{\prime }(T_i)$$

$$D_S^{\prime }(S_j)$$

$$F_{AI}(T_i)$$

$$F_{AI}(S_j)$$

$$W_d(S_k,S_l)$$

Full text

††terms: Algorithms, Experimentation, Performance

1 Introduction

Coevolution offers an approach to adaptively select tests for the evaluation of learners [9, 19, 11, 18, 17, 4]. Using coevolution, the evaluation function is adapted as part of the evolutionary process. This approach can be useful if the quality of individuals can be assessed using some form of tests. For such test-based problems, the identification of an informative set of tests can reduce the amount of required computation, while potentially providing more useful information than any static selection of tests. Since an adaptive test set can render evaluation unstable, an important question is how coevolution can be set up to be sufficiently reliable.

A recent insight in coevolution research is that the design of a coevolutionary setup should begin with a consideration of the desired solution concept [5]. A solution concept specifies which elements of the search space qualify as solutions and which do not. Examples of solution concepts include: Maximum Expected Utility (maximizing the expected outcome against a randomly selected opponent, which for uniform selection is equivalent to maximizing the average outcome against all opponents), the Pareto-optimal set resulting from viewing each test as a separate objective, and Nash-equilibria, in which no candidate solution or test can unilaterally deviate given the other candidate solutions and tests without decreasing its payoff.

For several of the main solution concepts used in coevolution, archive methods exist guaranteeing that when sufficiently diverse sets of new individuals are submitted to the archive, the archive will produce monotonically improving approximations of the solution concept. Recent examples of such archive methods are the Nash Memory [5, 8], which guarantees monotonicity for the Nash equilibrium solution concept; the IPCA algorithm, which guarantees monotonicity for the Pareto-optimal equivalence set [2]; and the MaxSolve algorithm [1], which guarantees monotonicity for the Maximum Expected Utility solution concept.

While theoretical guarantees of monotonic progress are important, so far no bounds or guarantees regarding the improvement of the approximation to the solution concept over time are available. Thus, approximating the solution concept to a desired degree of accuracy may take an infeasible amount of time. An important current practical question is therefore: how can coevolutionary algorithms be set up such that their dynamics lead to quick improvement over time? By using such efficient algorithms as generators of new individuals and coupling them to monotic archives, thereby combining the guarantee of monotonic progress with efficiency, a principled approach to designing robust and efficient coevolution algorithms is obtained.

Efficiency in coevolutionary algorithms depends on selection, see e.g. [6], and evaluation. In this research we focus on evaluation. Our aim is to compare the efficiency, reflected in the improvement over time of an objective quality measure, that can be achieved using different coevolutionary evaluation methods. Since a main question is how sufficiently accurate evaluation may be achieved, the testing environment is chosen such that evaluating individuals on all tests is feasible; while this is not the case in practical applications of coevolution, this provides a possibility to compare evaluation methods with the maximally informative situation in which information about all possible tests is available. This setup permits investigating two important questions:

Given all information that may be relevant to evaluation, how can this information be used optimally? 2. 2.

Compared to evaluation based on all relevant information, how do different coevolutionary evaluation methods perform?

In this paper, we focus on the second question. Four different coevolutionary evaluation methods are compared to each other and to the baseline of testing against all tests. The test problem is a small variant of the Majority Function test problem [16] [15] [11] chosen such that evaluation against all test cases (initial conditions) is feasible. A new tool named the Objective Fitness Correlation (OFC) [3], the correlation between the subjective and the objective fitness measures, is used to assess the evaluation accuracy of the different methods.

The paper is structured as follows. In section 2 we discuss the evaluation methods and algorithms used in this research. The design of experiments, parameters and performance measures are described in section 3. The results are presented in section 4, and the discussions and concluding remarks are shown in section 5.

2 Evaluation Methods

In this section, we describe the evaluation methods conducted in this study. Since we put our main focus on the test-based problems, we start with defining some terminologies and fitness measures. Evaluation methods based on those fitness measures are introduced in the following sections.

2.1 Interaction matrix

For a test problem, there are two sample spaces: one is the test cases, $T$, and the other is possible solutions, $S$. The term interaction is defined as the result of letting one solution interact with one test case. For the ease of analysis, an interaction function, $G(T,S)$, is designed to return one scalar outcome of the interaction between a pair of test case and a solution. To simplify the following discussion, we assume that the interaction function returns a binary result which represents whether the solution succeeds in solving the test case or not. An affirmative interaction is preferred by the solution, but is unfavorable to the test case.

By evaluating all solutions, ${\bf S}=\{S_{j}|j=1,2,....,m\}$, against all test points, ${\bf T}=\{T_{i}|i=1,2,....,n\}$, an interaction matrix, $I=\{I_{ij}|i=1,....,n;j=1,....,m\}$, is obtained. The sum of column $j$, $I_{j}$, of this interaction matrix represents the number of test cases that $S_{j}$ has solved, which is a common performance measure for a solution. Similarly, the sum of row $i$, $I_{i}$, is the number of times this test case $T_{i}$ being solved, which represents the difficulty of the test case.

In the simplest case, only the interaction outcomes are considered, so that $-I_{i}$ can be used as the fitness of $T_{i}$ (i.e., the more difficult the better), and $I_{j}$ for $S_{i}$ (i.e., the more powerful the better).

2.2 Distinctions

Distinctions are defined as the ability to distinguish between good and bad solutions and may provide important information for selecting a proper set of test cases. This concept was first proposed in [7], and here we follow the notation in [4] for our experiments. Accordingly, a three-dimensional matrix $dist(T,S,S)$ is defined to represent ”if test case $T_{i}$ distinguishes $S_{k}$ from $S_{l}$”, or mathematically:

[TABLE]

where $T_{i}\in{\bf T}$ and $S_{k},S_{l}\in{\bf S}$.

The number of distinctions that test case $T_{i}$ has made may be represented by summing up all $m^{2}$ solution pairs in $dist$, i.e., $dist_{T}(T_{i})=\sum_{kl}dist(i,k,l)$. This value represents the ability of a test case to maintain diversity of the solutions, and it may also be considered as a fitness measure for test cases.

Although we explain the concept of distinction in terms of “selecting a proper set of test cases”, the same procedure can also be applied to solutions. This yields the distinction of solutions, and is also a reasonable fitness measure for evaluating solutions. With those measures, four different evaluation methods are defined as follows.

2.3 Four evaluation methods

For each evaluation method, we define the fitness for test cases $F(T)$ and solutions $F(S)$ as follows.

2.3.1 Average score

For a solution, the average-score is defined as “the proportion of test points it has succeeded in solving.” And for a test point, this value is represented by ”the proportion of solutions that it has failed”. With an interaction matrix $I(i,j)$ as described earlier, we define:

[TABLE]

The application results of $F_{AS}$ on test cases and solutions are both between $[0,1]$, and a higher value represents a better performance.

2.3.2 Weighted score

Instead of giving each interaction an equal weight, the average scores described above may be used as weights for each test case and solution. That is to say, a test case earns more credits by failing a more powerful solution, and a solution gets a higher score when it succeeds in a tough test case. The mathematical expression of this evaluation can be represented as follows:

[TABLE]

Weights are checked to avoid being divided by zero and normalized to ensure the weighted scores are ranged from [math] to $1$. Hence $F_{WS}$ also returns a value between $[0,1]$, and the higher values are more preferable. This definition is similar in spirit to the niching methods of Rosin [19] and Juille [10], though their methods are not considered while we developed the weight function.

2.3.3 Average informativeness

The informativeness of a test measures the amount of information it provides about a given set of candidate solutions. In [bucci:foga], a definition for informativeness based on the incomparable and equal elements of the order induced by a test is provided. Here, we measure the informativeness of a test based on Ficici’s notion of distinctions [7]. Since each distinction a test makes contributes to its informativeness, and since the set of all possible distinctions is sufficient to provide ideal evaluation [4], we measure the informativeness of a test as the normalized number of distinctions it makes. In this study, we define the distinction score, $D$, as “the number of distinctions one test case (solution) makes.” This value can be derived from the distinction matrix described in (1), and represents the informativeness. For the convenience of further computation, we normalize the distinction score with its maximum and minimum values, which are returned by the function $Max(D)$ and $Min(D)$. Because the informativeness represents the individual’s ability to maintain diversity of the other population, we want to integrate it with the average scores to form the fitness. Here we simply use a linear combination of two scores, with weights of $(0.3,0.7)$ for distinction and average score, respectively. These weights are based on the observation that the average scores are generally lower than the distinction scores due to the computational scheme we used, as well as experiences from the pilot experiments. The average informativeness is therefore defined as the linear combination of the normalized distinction score and the average score.

[TABLE]

2.3.4 Weighted informativeness

Similar to the weighted score, each distinction that has been made can be weighted differently when the distinction score is derived. For example, the distinction made by all test cases, $\sum_{i=1}^{n}dist(T_{i},S_{k},S_{l})$, provides the information of “how many test cases have made the distinction on $S_{k}$ and $S_{l}$, such that $G(T_{i},S_{k})>G(T_{i},S_{l})$.” The inverse of this value can be used as the weight of this particular distinction, so that when more test cases can make this distinction, the less worthy this distinction is. This operation can also be applied to solutions, and the weighted informativeness can be defined mathematically as:

[TABLE]

In the following discussion, the four evaluation methods described above are referred as AS, WS, AI, and WI, respectively.

2.4 Algorithms for experiments

There are two main algorithms used in this study, i.e., a single population genetic algorithm (GA) and coevolution (CO). The former uses only one single population for solutions and evaluates the population on all possible test cases to obtain the interaction defined earlier. Afterward, the average scores given by $F_{AS}$ are used as the fitness in the GA. The results of this algorithm are used as the baseline for comparisons. The second algorithm uses coevolution between test cases and solutions, and all four evaluation methods are tested.

Algorithm 1 and 2 describe GA and CO, respectively. Functions used in the algorithms, e.g., INTERACTION, EVALUATE, SELECT and BREED, are specific to problems and experiments. The general concepts of INTERACTION and four implementations of EVALUATE are already discussed in this section. The SELECT and BREED together define the selection, reproduction and replacement cycle in a generation. The choice of these two functions is problem specific and they are specified in the experimental design section.

3 Experimental Setup

In this section, the settings of our experiments are described. The software used in this research is implemented as an extension of ECJ [14], which is developed by George Mason University’s Evolutionary Computation Laboratory. All simulations use the basic evolutionary loop provided by ECJ, plus the evaluation methods and problem-specific functions in the extension. The design of the experiments, the parameters used in the test problem, and the performance measures are introduced as follows.

3.1 Design of experiments

As mentioned in the previous section, two algorithms and four evaluation methods are used in this study. Table 1 shows the design of our experiments. Experiment 1 is a combination of the single population genetic algorithm and the average score evaluation method. This GAAS evaluates the solutions against all possible test cases and serves as the baseline experiment. Experiment 2, 3, 4, and 5 are four different evaluation methods combined with the coevolution algorithm. Each experiment is run for ten times with ten different random seeds.

3.2 Parameters for the test problem

In this study, we perform the experiments on the majority function problem. This problem is also known as the one dimensional cellular automata problem. Mitchell and his colleagues have detailed discussion on this problem in [16] and [15]. Two major parameters are used for this problem, i.e., the radius of the neighbourhood ($r$) and the size of the one dimensional lattice ($N$). Boolean vectors are used to represent both the initial conditions (test cases) and the rules (solutions). In order to evaluate against all test points, we chose $r=2$ and $N=9$. Other parameters, e.g., the size of populations, elitism, crossover type, and mutation probability are selected based on Mitchell’s work [15], and are summarized in Table 2.

In the coevolution experiments (2, 3, 4, and 5), a symmetric setup is used, i.e., the same set of EVALUATE, SELECT and BREED functions are used for both population of test cases and the population of solutions. However, there is one exception: the population size. Because there are only 512 possible test cases in total, the population size for test cases is set as 64 in the convenience of comparison, while the population size for solutions is set as 100. Accordingly, the number of elites for two populations are also different, while their proportions to the size of populations are the same (20%). The initial populations are both randomly created and are both checked to ensure there are no redundent individuals in the begining of the experiments.

3.3 Performance measures

In order to compare all experiments fairly, an objective performance measure is required. Hence we define the subjective fitness and objective fitness separately. The subjective fitness is the fitness value returned by the EVALUATE function used in the coevolution algorithm. In this study, the objective fitness is defined as the fitness used in experiment 1, i.e., to evaluate the average scores against all possible test cases. For each generation in experiment $2\sim 5$, both the subjective and the objective fitness are recorded. All experiments are compared with the their objective fitness, based on the number of interactions. In coevolution experiments, the number of test cases is $1/8$ of that in the GAAS experiment, therefore during the analysis we compare the objective fitness of the coevolution experiments every $8$ generations to the fitness of the controlled experiments every generation. Since these values are all derived from exhaustive testing, the progress over interactions may be seen as the performance of each method.

In addition to the progress of the fitness, the correlation coefficients between subjective and objective fitness are also computed. This correlation is defined as the Objective Fitness Correlation, OFC, in [3] as a new objective measure for evaluating coevolutionary algorithms. In our experiments, this measure is collected for each generation of every run. Since OFC is always $1$ in the baseline experiment, we only compare OFC among the four evaluation methods with coevolution. Comparisons are made with these two measures averaged upon ten independent runs.

4 Results

4.1 Best objective fitness

Figure 1 shows the best objective fitness of the solutions averaged over 10 runs. As illustrated in the figure, the baseline experiment is outperformed by all coevolution experiments, especially when the number of total interactions is small. Among four evaluation methods with coevolution, the weighted informativeness shows a constantly better performance than others.

A series of paired, one-tailed t-tests are performed to examine the significance of the differences between each evaluation methods. The best fitness of each generation is averaged over 10 runs, and is paired with the same quantity of another evaluation method for the t-test. The results of t-tests shows significance on WI $>$ AI, WI $>$ WS, WI $>$ AS, AI $>$ AS, and WS $>$ AS (with p-values $<1\times 10^{-10}$), but not on AI $\neq$ WS with $\alpha=0.05$. These results are summarized in table 3.

4.2 Objective fitness correlation

The OFCs for different evaluation methods with coevolution are shown in figure 2. The values shown in the figure are averages of 10 independent runs, and therefore the fluctuations of OFCs are already smoothed. In a single run of one experiment, the OFC can be negative for some generations.

In all experiments, the OFC always starts from a high value. This is because the initial populations are randomly generated, and this can be seen as they are sampled from the set of all possible cases with a uniform distribution. Ideally, this random sampling may create a set well represents the original search space, i.e., the all possible cases, and results in a higher correlation between the subjective and the objective fitness. As the coevolution proceeds, the populations move toward certain direction rather than a uniformly random sampling, and hence the OFC decreases over generations. However, as argued in [3], the OFC may still remain as a performance measure for comparing different evaluation methods in a coevolutionary algorithm.

As demonstrated in figure 2, the weighted informativeness shows a constantly higher OFC through generations than other methods, while other evaluation methods do not show clear differences between one another due to the fluctuations.

A set of t-tests is employed to verify the difference in OFC between each pair of evaluation methods. The results show significance on all of following relations: WI $>$ AI, WI $>$ WS, WI $>$ AS, AI $>$ WS, AI $>$ AS, and WS $>$ AS. The preferences for WI against other methods are very significant, with p-values $<1\times 10^{-15}$, which is the limit of precision in the computing software. These results are summarized in table 4.

5 Discussion

5.1 Evaluating with the weighted informativeness

The results of our experiments suggest that in coevolution, a combination of performance (average score) and diversity (weighted distinction) can achieve the accuracy of full evaluation with less computational cost. This statement holds for both conditions when considering the improvement of an objective quality measure over time or the correlation between the subjective fitness and objective measure. This advantage is even clearer if we compare the objective fitness according to the generation instead of the number of interactions. As shown in figure 3, the WI evaluation method with coevolution progresses as fast as evaluating against all possible test cases, while other evaluation methods are apparently slower than the baseline.

In our study, the OFC is used as a quality measure in addition to the objective fitness. If the preference orderings of evaluation methods are considered, this measure provides information similar to the objective fitness. However, the difference between WS and AI shows significance in OFC but not in the objective fitness. This implies that two measures may contain different information, and a detailed discussion on OFC can be found in [3].

5.2 The majority function problem with different sizes

This research is the first time the OFC is calculated on a real problem, but in exchange we are only able to experiment on problems with a small number of possible test cases in total. The majority function problem with different sizes ($N=7$ and $N=11$) are also tested. The results are similar to the finding presented in the previous section, and the experiments of $N=11$ are summarized in figure 4 and 5. From these figures it is shown that OFC can distinguish AI and WI from AS and WS while the objective fitness can not. This argument is consistent with the results of t-tests (not shown): t-tests on the objective fitness show no significant difference between pairs among AS, WS, and AI, while those on OFC show the same preference as in the $N=9$ experiments.

5.3 The parity problem

In addition to the majority function problem, the same comparison is also performed on the $n$-odd-parity problems. Although Lee, Xu, and Chau [13] have proposed that the boolean parity problem may be transformed into a cellular automata problem, here we use Koza’s Genetic Programming approach in [12].

Figure 6 and 7 show the results for the $6$-odd-parity problem. As shown in figure 6, the WI still performs the best among four evaluation methods in coevolution, and this is also confirmed by the t-tests (not shown). However, we also find some disagreement between results from the majority function problem and the parity problem. First, in the parity problem, the single population GA outperforms all CO experiments in this genetic programming approach, while figure 1 shows the opposite in the majority function problem. Second, OFC in the parity problem shows no significant pattern (as shown in figure 7), while the OFC distinguishes different evaluation methods better than the objective fitness in the majority function problem. Finally, the t-tests hardly show any significant difference among AS, WS, and AI in the parity problem.

A detailed analysis on the output has been done and suggests a few possible reasons for the disagreement. First, solving the parity problem with the genetic programming approach is not a symmetric test-based problem in nature. That is to say, while the test cases are represented with boolean vectors as they are in the majority function problem, the solutions for the parity problem are represented with tree-like structures. As a result of this asymmetry, the same evolutionary operator (e.g., selection methods, mutation rate, crossover methods, etc.) may have different effects on two populations, and hence a symmetric setting for coevolution is not suitable. Second, the initial population of the tree-like solutions is not randomly selected from “all possible solutions”. In Koza’s approach, the sizes of the solutions start from smaller values and grow over generations. This “biased sampling” may explain the disfavor of OFC in the parity problem. Finally, since Koza’s approach has been studies for several years, the evolutionary operators and parameters are already well tuned. We believe that the disagreement may be reduced by developing proper asymmetric coevolution schemes.

Despite the disagreement, the results of the parity problem still shows a favor to WI in coevolution.

6 Conclusion

In this research, we compare four different evaluation methods in coevolution on a test-based problem. Two measures are used for the comparisons among average score (AS), weighted score (WS), average informativeness (AI), and weighted informativeness (WI). In addition to an objective quality measure, the objective fitness correlation (OFC) is also computed.

The experimental results show a strong preference on WI, which suggest that a combination of the performance and the ability to create distinctions may provide more accurate evaluation in coevolution. The resulting significance from t-tests show a similar preference when two quality measures are used, separately. This study also uses the recently proposed OFC to evaluate the accuracy of coevolutionary evaluation methods on a concrete test problem.

Although we have shown the advantages of using WI in coevolution, the way we combine these two measures is simply using a weighted summation. It may worth exploring more sophisticated methods to fuse this information together. Currently a multi-objective approach is in progress, and this may lead to a more detailed investigation on how to use both measures in coevolutionary algorithms.