Hierarchical Association Coefficient Algorithm
Bongsong Kim
TL;DR
The paper introduces a new algorithm to measure how ordered data fits within stratified categories.
Contribution
The Hierarchical Association Coefficient algorithm is proposed to quantify the proximity of ordered data to ideal sorting extremes.
Findings
The HA-coefficient algorithm is grounded in theoretical principles of stratified categorization.
Example data and simulations demonstrate the algorithm's practical utility and reliability.
The algorithm can assess how close an observed order is to an ideal ascending or descending extreme.
Abstract
Suppose that members in a universal set categorized based on observations, and that categories can be stratified based on the average of observations within each category. Two sorting extremes can be obtained from the perspective of arbitrariness of an order of observations. The first sorting extreme is an increasing order of observations on ascendingly stratified categories. The second sorting extreme is a decreasing order of observations on ascendingly stratified categories. Hierarchical association coefficient (HA-coefficient) algorithm is based on a principle that any order of observations in stratified categorization can be placed between the two sorting extremes. The algorithm produces a proportion of how much an order of observations in stratified categorization is close to the first sorting extreme, or how much an order of categorized observations is distant from the second…
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRough Sets and Fuzzy Logic
Introduction
Hierarchical association coefficient algorithm (HA-coefficient algorithm) can work with observations in stratified categorization based on the average of observations within each category. The algorithm is fundamentally based on a concept that any marginal data between observations and the categorical identifiers is one outcome out of all possible permutations of the observations against categorical identifiers. From any marginal data between observations and categorical identifiers, two sorting extremes can be obtained by arranging the observations and categorical identifiers in the following manner: (1) the observations are increasingly sorted into ascendingly stratified categories; (2) the observations are decreasingly sorted into the same categories. The algorithm produces a proportion for arbitrariness of an order of observations given ascendingly stratified categorization between the two sorting extremes, and the proportion is called hierarchical association coefficient (HA-coefficient). Thus, the former and latter sorting extremes always produce the HA-coefficient = 1 and HA-coefficient = 0, respectively, so that the HA-coefficient indicates how much the observed order of observations is close to the increasing order of observations given ascendingly stratified categories. The algorithm measures the HA-coefficient in a physical manner based on distance and area.
This paper introduces the HA-coefficient algorithm, and shows its applications with three different situations. Also, it presents a demonstration that the algorithm produces a consistent result with three different simulated data sets that actually share the same pattern across the simulations. This result proves that the algorithm is reliable. The development of HA-coefficient algorithm was initially motivated to detect single nucleotide polymorphisms (SNPs) of high association with traits of interest in soybean breeding. As many disciplines need an analysis of data in stratified categorization, the HA-coefficient algorithm can be implemented to a wide spectrum of studies.
Theory and method
Hierarchical association distance metric
Assume that data satisfy the following conditions:
- Every member has a positive real value as observation.
- Every member has either nominal or ordinal identifier that determines its own category.
- All categorical averages of observations are different.
Under the above conditions, the observations and categories for every member are marginal. And, multiple categories can be stratified based on the average of observations within each category. Based on these aspects, Definitions 1 to 4 were established:
Definition 1.“Hierarchical” defines that categorical groups are stratified based on the average of observations within each category.
Definition 2.Suppose that multiple categories are sorted in ascending order based on the average of observations within each category. “Top categorization” defines a condition that observations arranged in ascending order within each category lead to ascending order across the multiple categories.
Definition 3.Suppose that multiple categories are sorted according to the top categorization according to Definition 2. “Bottom categorization” defines a condition that observations arranged in descending order within each category lead to descending order across the multiple categories.
Definition 4.“Hierarchical association” defines a proportion representing how close the top and observed categorizations are, or how distant the bottom and observed categorizations are.
In Definition 1, “hierarchical” attribute is a crucial concept since all computations for the HA-coefficient algorithm require categories to be sorted in a hierarchical manner. Note that permutation of observations within a category does not affect a calculation associated with the hierarchical association. Suppose that multiple categories are hierarchically arranged in a universal set. The set can be divided into two subsets at any categorical boundary. For convenience, say that the left and right subsets were obtained, denoted by L and R, respectively. If the sum of the L subset in the top categorization is greater than that in the observed categorization, say that L is Category 1 and its sum is x1, and that the R is Category 2 and its sum is x2. Likewise, R and L can be Categories 1 and 2, respectively.
Definition 5.Suppose that multiple categories and observations are marginal and hierarchically stratified in a universal set. The set can be divided into two subsets (S1 and S2) at any categorical boundary. A sum of either S1 or S2 must be greater in the top categorization than in the observed categorization, which is defined by Category 1, and its sum is defined as x1. Thus, the other category and its sum are defined by Category 2 and x2, respectively.
Definition 5 defines a rule to determine the x1 in Category 1. As a component of HA-coefficient algorithm, the hierarchical association distance metric (HADM) calculates a d coefficient. Say that the d coefficient given a variable x is denoted by dx, where a subscript x on dx refers to x1 according to Definition 5. And say that g1 and g2 are x1 and x2, respectively, in the top categorization according to Definition 5, and that r1 and r2 are x1 and x2, respectively, in the observed categorization according to Definition Definitions 5. The dx can be can be calculated as: [eqn] where x = the variable; dx = the d coefficient given x; y = the sum of all observations; g1 = the x1 in the top categorization according to Definition 5; g2 = the x2 in the top categorization according to Definition 5.
The Equation 1 can be derived: [eqn]
Substitute r1 by x as a variable, so that r2 = y − x. Note that x is a variable referring to x1. Then, [eqn] where r1 = the x1 in the observed categorization; r2 = the x2 in the observed categorization; x = the variable; dx = the d coefficient given x; y = the sum of all observations; g1 = the x1 in the top categorization according to Definition 5; g2 = the x2 in the top categorization according to Definition 5.
It is always true that [eqn] and [eqn], so that 1 ≤ dx. The top categorization according to Definition 2 always produces dx = 1 as a minimum d coefficient, while the bottom categorization according to Definition 3 always results in maximum dx. The dx can be interpreted as a hierarchical association distance given x = x1 from any categorization. In dealing with dx, the categories 1 and 2 have to be hierarchical according to Definition 1. If they are not hierarchical, dx is unsolvable.
HA-coefficient algorithm with two categories
The dx against x in the HADM (Equation 1) is shaped into a curve (see Figure 1). An area delimited between two x1 corresponding to the bottom and top categorizations refers to cumulative hierarchical association distances from all possible permutations of the observations against marginal categorical identifiers, and say the area as W. Likewise, an area delimited between two x1s corresponding to the bottom and observed categorizations refers to cumulative hierarchical association distances between the two categorizations, and say the area as R. The W and R can be calculated as follows: [eqn] [eqn]
A curve for . This shows a curve of dx against x given the first boundary of SNP1 data in Table 1 (see Table 2 also), in which x-and y-axes represent scales for x and dx, respectively. The all area (W) and realized area (R) can be determined with area-in and area-out under the curve delimited. The graph was generated using online graph calculator at https://www.desmos.com/calculator.
Ultimately, the HA-coefficient can be obtained as a ratio between R and W as follows: [eqn] where HA = the HA-coefficient; x = the variable; W = the whole area that x can formulate between two x1s corresponding the bottom and top categorizations; R = the realized area that x can formulate between two x1s corresponding the bottom and observed categorizations; obs. = the x1 in the observed categorization; top. = the x1 in the top categorization; bot. = the x1 in the bottom categorization; y = the sum of whole observations; g1 = the x1 in the top categorization; g2 = the x2 in the top categorization.
It is always true that 0 ≤ R ≤ W, so that the HA-coefficient results in a proportion. The HA-coefficient represents the hierarchical association according to Definition 4. So far, two categories of data were addressed. In reality, it is general that observations can be classified into more than two hierarchical categories. The calculation of HA-coefficient can be extended to data with any number of hierarchical categories.
HA-coefficient algorithm with any number of categories
In case that two or more categories have equal averages, it should be carefully considered that the categories can be merged into a single category. If the categories cannot be merged due to other attribute, dx is unsolvable because the categories are not hierarchical according to Definition 1. In case that a universal set nests n categories, the set can be classified into two subsets of n-1 at each categorical boundary. With multiple categories, the HA-coefficient algorithm can be implemented following two procedures: (1) computing component HA-coefficients of n-1 times at each categorical boundary, and (2) averaging all component HA-coefficients. As averaging method, either of harmonic and arithmetic means can be taken. The methods for computing the HA-coefficient can be denoted as follows:
- HA-coefficient algorithm based on harmonic mean, [eqn]
- HA-coefficient algorithm based on arithmetic mean, [eqn] where HA = the HA-coefficient; x = the variable; n = the total number of categories; k = the variable referring to an order of categorical boundary; y = the sum of all observations; bot.[k] = the x1 in the bottom categorization given the k’th categorical boundary; obs.[k] = the x1 in the observed categorization given the k’th categorical boundary; top.[k] = the x1 in the top categorization given the k’th categorical boundary.
Both Equations 5 and 6 produce identical results, and are complete forms of the HA-coefficient algorithm. In implementing the algorithms, classifying a universal set into two subsets of n-1 times at each categorical boundary provides the following properties:
- computing the component HA-coefficients at each categorical boundary happens with entire data, which allows to obtain an ultimate HA-coefficient by simply averaging;
- any change of dx from permuting observations against categorical identifiers can be captured through discovering a change of a ratio between x1 and x2.
Independent categories and dependent categories
The categorization can be hierarchical in pre-determined (pre-hierarchical) or post-determined (post-hierarchical) manner. If categories are determined independent from the observations, categories are pre-hierarchical. If categories are determined dependent on the observations, categories are posthierarchical.
Definition 6.When categorical hierarchy is determined independent from observations, categories are “prehierarchical”. When categorical hierarchy is determined dependent on observations, categories are “posthierarchical”.
It is important to note that Definition 6 determines a feasibility that HA-coefficient = 0. The independent categorization makes it feasible that HA-coefficient = 0, while the dependent categorization does not make it feasible that HA-coefficient = 0. This will be addressed in detail across the three example situations in Results and Discussion.
Curve of dx
Figure 1 shows a part of curve for [eqn], which is derived from SNP1 scores given soybean yield quantity at the boundary between M0 and (M1, M2) in Table 1 (see Table 2 also). On the graph, x-axis and y-axis represent a scale of x and dx, respectively. The dx curve can be used to determine both the W and R with setups of both area-in and area-out. On the graph, the x can always have a value between two x1s responding to the bottom and top categorizations. Note that the x can also be equal to either of x1s responding to the bottom and top categorizations, which will result in HA-coefficient = 0 or HA-coefficient = 1, respectively. As shown in Figure 1, the dx curve can be drawn only in quadrant I, which indicates that only positive real numbers are available as observations.
Table 1.: Simulated SNPs’ scores and yield quantity (kg / ha) for 20 soybean varieties. At the bottom of table, rounded means within each category are summarized, and its hierarchical ranking is shown in parentheses.
Table 2.: Summary of x1 and x2 extracted from Table 1. The x1 and x2 were determined at each categorical boundary according to Definition 5. By applying the HA-coefficient algorithm to the x1s, the HA-coefficient can be obtained. Hierarchical ranking of categories is ascendingly present within parentheses underneath each SNP name in the first column. The second column shows classifications of the whole set using a vertical bar and parentheses, and x1 and x2 are marked within square brackets. As all the three SNPs have three categories, in common, the whole set can be classified at two different categorical boundaries at a time for each SNP.
Data generation for simulations
Four simulated data sets were generated using R package (R Core Team, 2014). Of these, three were used to show an application of practice the HA-coefficient algorithm, and the other three were one was used to prove a reliability of the algorithm. The first three data former sets were manually adjusted after generation.
Simulations to prove a reliability of HA-coefficient algorithm
In order to see if the HA-coefficient algorithm is robust, three simulated data sets in simple design were generated. The Respective simulated data sets have two to four categories. In the simulation, the following two questions were focused on, which are:
- Do the resulting HA-coefficients gradually increase as categorizations change from arbitrary to top?
- Does the HA-coefficient algorithm produce the same result for three different shapes of data that are inherently in the same pattern?
As an example, the simulated data set where three categorical identifiers and observations are marginal was made in the following steps:
- Create a vector consisted of 1,200 numbers from 3 to 3,600 at an interval of 3 in ascending order. These values will be used as observations.
- Create a 1,200 by 1,200 matrix filled with categorical identifiers, 0, 1, and 2 generated randomly.
- Divide the matrix equally into three areas, vertically. Subsequently, overwrite 0, 1, and 2 on the area above the diagonal at each of the third, respectively.
- Append the vector for the observations to the right end of the matrix. By doing so, the matrix size becomes 1,200 by 1,201.
- Create a repository vector that will contain the 1,200 resulting HA-coefficients.
- Calculate the HA-coefficient with a pair of columns for identifiers and observations across the matrix of identifiers.
- Append the resulting HA-coefficients to the repository vector.
- Repeat the above steps 100 times, and average the 100 repository vectors.
Above procedures illustrate generating the simulated data that have three categories. The simulated data carrying the two and four hierarchical categories were prepared in a manner of dividing the entire matrix into two and four sub-areas, respectively. The simulated data sets can be seen in Figure 2. The designs of the simulations intend to output gradually increasing HA-coefficients as a column of the matrix changes from left to right. In order to draw a smooth plot, the entire cycle of the simulation was repeated 100 times, and a plot was obtained through averaging.
Three simulated data sets. (a), (b), and (c) are filled with categorical identifiers of two-, three-, and four-types, respectively. Each square includes multiple blue triangles. Within each square, the same identifiers are filled in each blue triangle, but different blue triangles have different identifiers. Yellow triangle contains arbitrary categorical identifiers, and green triangle on the right side refers to a vector containing 1,200 observations from 3 to 3,600 at an interval of 3. When a pair of columns for the left end of the matrix and observations are marginal, arbitrary categorization will be obtained, while the top categorization will be formed with a pair of columns for 1,200th column and observations.
Results and discussion
Let us apply the HA-coefficient algorithm to the following three example situations. The three examples refer to varying situations based on a dependency between categories and observations according to Definition 6. Situation 1 presents that categories are dependent on observations, while Situations 2 and 3 show that categories are independent from observations. Situation 3 is an example that categories are determined by multi-dimensional observations.
Situation 1: Categories fully depending on observations
Table 1 displays simulated genotypes and phenotypes, which are categorized based on singlenucleotide polymorphisms (SNPs) and observations for yield (kg/ha), respectively. The SNPs classify 20 soybean varieties into three categories (0, 1, and 2), and a hierarchical level among categories can be determined based on the average of yield within each category. This is a typical post-hierarchical case according to Definition 6 since the hierarchical categories are obtained depending on observations. This condition makes it impossible to arrange an order of observations to be in the bottom categorization since the hierarchical ranking of the categories can be altered with permuting the observations against categorical identifiers. Once categories are fixed based on the observations, the bottom and top categories can be determined. The scores from each SNP were resorted according to the top and bottom categorizations, which are tabulated in Supplementary table 1. By referring to the resorted observations, the x1s and x2s for each SNP can be determined according to Definition 5 at each categorical boundary. Table 2 shows a summary for both x1s and x2s for the three SNPs in Table 1. Basically, Table 2 shows a division of the total sum into two sums at each categorical boundary. Therefore, total sums are consistent across all boundaries, but divided sums vary depending on classification.
By applying Equation 5 to the x1s from Table 2, the HA-coefficients for the three SNPs were calculated as follows: [eqn] where HA_SNP1_= the HA-coefficient for SNP1; HA_SNP2_= the HA-coefficient for SNP2; HA_SNP3_= the HA-coefficient for SNP3; x = the variable.
From the above results, it was identified that SNP2 has the strongest hierarchical association, followed by SNP1 and SNP3.
Situation 2: Categories independent from observations
When hierarchical ranking of categories is given from an external source, it satisfies a pre-hierarchical condition that categories and observations are independent according to Definition 6. Table 3 shows the scores from two Mathematics quizzes for the same 27 students. Based on the scores from Quiz 1, a teacher classifies all students into three hierarchical categories upon the following criteria: [eqn]
Table 3.: Simulated scores of two Mathematics quizzes for 27 students. The three categories were classified based on the scores from Quiz 1, in which Categories 3, 2, and 1 satisfy conditions: 70 ≤ score, 60 ≤ score < 70, and score < 60, respectively.
If the ranking of scores from Quiz 2 is preserved or exchanged within each category based on Quiz 1, there will be no difference on hierarchical association between results from the two quizzes. However, new scores from different quiz for the same students are often differently ranked across categories, so that the teacher could raise the following questions. Was the categorical stratification based on Quiz 1 maintained in the result from Quiz 2? If not, how much of categorical solidity based on Quiz 1 remains in the result from Quiz 2? In order for the HA-coefficient algorithm to answer this question, the categorization determined based on Quiz 1 has to be fixed, and applied to the result from Quiz 2. The scores from Quiz 2 were resorted according to the top and bottom categorizations, which are tabulated in Supplementary table 2. Table 4 presents the x1s and x2s at each categorical boundary. The HA-coefficient were calculated by applying Equation 5 to the x1s from Table 4 as follows: [eqn] where HA = the HA-coefficient; x = the variable.
Table 4.: Summary of x1s and x2s extracted from Table 3. The first column shows hierarchical ranking of three categories (C1, C2, and C3) and classified sets (x1 and x2), in which a vertical bar is a boundary to classify the whole set, and parentheses collapse categories into the same set.
The above result indicates that the result from Quiz 2 keeps 71.5 % of solidity of categorization determined by Quiz 1.
Situation 3: Categories determined by multi-dimensional observations
It is possible that multi-dimensional observations form a categorization. In this case, each dimension of observations influences the categorization. Table 5 displays simulated data about results from Mathematics and English quizzes. The table includes Mathematics scores, English scores, and average for the same 27 students, and the students are hierarchically categorized based on the average. Thus, the Mathematics and English scores are related to the categorization. The criteria for the categorization are as follows: [eqn]
Table 5.: Simulated scores of Mathematics and English quizzes for 27 students. The three categories were classified based on averages, in which Categories 3, 2, and 1 (C3, C2, and C1) satisfy conditions: 70 ≤ average score, 60 ≤ average score < 70, and average score < 60, respectively.
The size of each category and categorical classification of each student are determined based on the averages, so that this situation is pre-hierarchical. If the ranking from each quiz is preserved or exchanged within each category based on the averages, there will be no difference on hierarchical association between each quiz and average. However, the scores from the two quizzes ranked across categories, so that the following questions could be raised. Was the categorical stratification based on the averages maintained in the result from each quiz? If not, how much of categorical solidity based on the averages remains in the result from each quiz? To answer the questions using the HA-coefficient algorithm, the given categorization has to be fixed, and applied to the scores from each quiz. The scores were resorted according to the top and bottom categorizations, which are tabulated in Supplementary table 3. Table 6 shows the x1s and x2s at each categorical boundary. By applying Equation 5 to the x1s from Table 6, the HA-coefficients were calculated as follows: [eqn] where HA_Math‒Average_ = the HA-coefficient for the Mathematics scores given the averages; HA_English‒Average_= the HA-coefficient for the English scores the given averages; x = the variable.
Table 6.: Summary of both x1 and x2 extracted from Table 5. The second column shows hierarchical ranking of the three categories (C1, C2, and C3) and sets classified into x1 and x2, in which a vertical bar is a boundary to classify the whole set, and parentheses collapse two categories into the same set.
The above results explain that the Mathematics and English scores keep 92.6 and 91.1 % of solidities of the categorization determined by the averages, respectively.
Simulation to evaluate a reliability of HA-coefficient algorithm
In order to verify a reliability of the algorithm, the simulation was conducted with the two following questions:
- As a categorization is getting from arbitrary to top, does the algorithm produce an increasing HA-coefficient?
- Does the HA-coefficient algorithm produce the same result for three different shapes of data that are inherently in the same pattern?
The above questions are very essential to evaluate the reliability of the algorithm. For the simulation, three data sets in square dimension (1,200 by 1,200) were generated. The first, second and third squares have two, three and four categorical identifiers within each column, respectively. Each simulated data set was vertically partitioned into two to four areas with equal size. And, a vector containing 1,200 numbers from 3 to 3,600 at regular interval of 3 was appended at the right end of each matrix. Figure 2 visualizes the structures of the simulated data. Within each square, each blue triangle is filled with the same categorical identifiers, but different blue triangles have different categorical identifiers. Yellow triangles refer to area for arbitrarily mixed categorical identifiers. Meanwhile, each figure has a green triangle with a slope, which implies an increasing pattern of observations. Thus, when pairing two columns for categorical identifiers and observations, an arrangement of categorical identifiers given the observations gradually transpositions from arbitrary to top as the column changes from left to right. A relationship between the categories and observations stays dependent to some degree of right column from the left end, but shifts to solid independent condition since averages for each category are gradually getting hierarchically solid as a column changes from left to right. The implementation of the HA-coefficient algorithm was repeated 100 times with each simulated data to minimize a noisy fluctuation. Figure 3 shows three outcome plots, illustrating a steady increasing pattern. This returns a positive answer to the first question. Meanwhile, the shapes of the three plots in Figure 3 seemingly look closely similar, and each plot ranges between around 0.75 and 1. Since the categorical identifiers and observations are post-hierarchical from the first column to some degree of depth to the right, these simulations do not allow to determine the bottom categorization same as explained in Situation 1. Therefore, it cannot happen that HA-coefficient = 0 with these simulations. Importantly, the three different simulations share two following properties: (1) equal proportions of identifiers according to the top categorization in column-wise comparison; (2) the identical observations in the same order. These two properties assure that the three different simulated data have the same pattern in a fashion of column-wise comparison. Therefore, in order for the HA-coefficient algorithm to be robust, the same results have to be obtained from the three simulations. Table 7 shows that the Pearson correlation coefficients among the three plots are remarkably close to one another. This returns a positive answer to the second question, and indicate that the averaging with n-1, where n is the number of categories, does not perturb calculating the HA-coefficient.
Plots (a), (b), and (c) are drawn based on the HA-coefficients resulted from the simulated data in Figure 2 (a), (b), and (c), respectively. Each plot was obtained through averaging 100 repetitions of implementing the HA-coefficient algorithm.
Table 7.: Pearson correlation coefficients among the three plots in Figure 3
Conclusion
Above, it was proven that the HA-coefficient algorithm returns a reliable result through the simulations. As long as the categories and observations are marginal, and categories are hierarchically stratified, the algorithm solves a reasonable proportion for HA-coefficient from data. It is important to note that the HA-coefficient is an objective measure rather than a statistical inference. The algorithm is easy to computerize, and its implementation does not require high computing power. The HA-coefficient algorithm was applied to real agricultural data including 4,312 SNPs and yield quantity for 5,180 soybean lines, its computation was done in 34 seconds on a laptop (Dell E6540) using R. This validates that the algorithm runs very fast on an ordinary computer. As many disciplines deal with data in hierarchical stratification, the algorithm will be useful for a wide spectrum of studies.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1R Core Team (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org/