On the relation between Loss Functions and T-Norms

TL;DR

This paper explores the connection between loss functions and t-norms, providing a new interpretation that leads to the development of novel loss functions potentially improving convergence in supervised learning.

Contribution

It introduces a general relation between loss functions and t-norms, offering a new perspective and a novel class of loss functions for supervised learning.

Findings

Derived a relation between loss functions and t-norms.

Proposed a new class of loss functions based on t-norms.

Potential for faster convergence than cross-entropy loss.

Abstract

Deep learning has been shown to achieve impressive results in several domains like computer vision and natural language processing. A key element of this success has been the development of new loss functions, like the popular cross-entropy loss, which has been shown to provide faster convergence and to reduce the vanishing gradient problem in very deep structures. While the cross-entropy loss is usually justified from a probabilistic perspective, this paper shows an alternative and more direct interpretation of this loss in terms of t-norms and their associated generator functions, and derives a general relation between loss functions and t-norms. In particular, the presented work shows intriguing results leading to the development of a novel class of loss functions. These losses can be exploited in any supervised learning task and which could lead to faster convergence rates that the commonly employed cross-entropy loss.

Tables

Table 1. Table 1 : Fundamental t-norms.

Gödel	Lukasiewicz	Product
$T_M(x,y)=\min \{x,y\}$	$T_L(x,y)=\max \{0,x+y-1\}$	$T_{\Pi }(x,y)=x\cdot y$

Equations

$$A_1\le A_2\text{mbox}iff{A}_1(x,y)\le A_2(x,y),\text{mbox}forallx,y\in [0,1].$$

$$T(x,y)=g^{-1}\left(\min \{g(0^{+}),g(x)+g(y)\}\right).$$

$$T(x,y)=1-\min \{1,1-x+1-y\}=\max \{0,x+y-1\}.$$

$$T(x,y)=e^{-(\min \{+\infty ,-\log (x)-\log (y)\})}=x\cdot y.$$

$$\text{mbox}residuum:$$

$$\text{mbox}bi-residuum:$$

$$g_{\lambda }^{SS}(x)=\{\begin{array}{ll}-\log (x)& \text{mbox}if\lambda =0\\ \frac{1-x^{\lambda }}{\lambda }& \text{mbox}otherwise\end{array}\text{mbox}and{g}_{\lambda }^F(x)=\{\begin{array}{ll}-\log (x)& \text{mbox}if\lambda =1\\ 1-x& \text{mbox}if\lambda =+\infty \\ \log \left(\frac{\lambda -1}{{\lambda }^x-1}\right)& \text{mbox}otherwise\end{array}$$

$$\mathcal{L}(\mathcal{X},\boldsymbol{S},\boldsymbol{p})=\sum_{h=1}^{H}\lambda_{h}L\big{(}f_{h}(\mathcal{X},\boldsymbol{S},\boldsymbol{p})\big{)}$$

$$\forall x_i\phi (x_i)$$

$$\exists x_i\phi (x_i)$$

$$\forall x\,\varphi(x)\equiv\displaystyle\bigwedge_{x}\varphi(x)\quad\Longrightarrow\quad g^{-1}\left(\min\left\{g(0^{+}),\sum_{x\in\mathcal{X}}g\big{(}f_{\varphi}(x,\boldsymbol{S},\boldsymbol{p})\big{)}\right\}\right)\ ,$$

$$\min \{g(0^{+}),\sum _{x\in \mathcal{X}}g(f_{\phi }(x,\mathbf{S},\mathbf{p}))\}$$

$$\sum _{x\in \mathcal{X}}g(f_{\phi }(x,\mathbf{S},\mathbf{p}))$$

$$\min (1,\sum _{x\in \mathcal{X}}(1-(f_{\phi }(x,\mathbf{S},\mathbf{p}))).$$

$$-\sum _{x\in \mathcal{X}}\log (f_{\phi }(x,\mathbf{S},\mathbf{p})).$$

$$\forall x{p}_1(x)$$

$$\forall x{p}_1(x)\Rightarrow p_2(x)$$

$$\forall x{p}_1(x)\iff p_2(x)$$

$$\forall x{S}_j(x)\iff p_j(x)⟹\mathcal{L}(\mathcal{X},\mathbf{S},p_j)=\sum _{x\in \mathcal{X}}|g(S_j(x))-g(p_j(x))|$$

$${\mathcal{L}}^{+}(\mathcal{X},\mathbf{S},p_j)=\sum _{x\in {\mathcal{X}}_j}g(p_j(x)),$$

$${\mathcal{L}}_L^{+}({\mathcal{X}}_j,p_j)=\sum _{x\in {\mathcal{X}}_j}\left(1-p_j(x)\right),{\mathcal{L}}_{\Pi }^{+}({\mathcal{X}}_j,p_j)=-\sum _{x\in {\mathcal{X}}_j}\log \left(p_j(x)\right)$$

Full text

1 Introduction

A careful choice of the loss function has been pivotal into the success of deep learning. In particular, the cross-entropy loss, or log loss, measures the performance of a classifier and increases when the predicted probability of an assignment diverges from the actual label [7]. In supervised learning, the cross-entropy loss has a clear interpretation as it attempts at minimizing the distribution of the predicted and given pattern labels. From a practical standpoint, the main advantage of this loss is to limit the vanishing gradient issue for networks with sigmoidal or softmax output activations.

Recent advancements in Statistical Relational Learning (SRL) [16] allow to inject prior knowledge, often expressed using a logic formalism, into a learner. One of the most popular lines of research in this community attempts at defining frameworks for performing logic inference in the presence of uncertainty. For example, Markov Logic Networks [18] and Probabilistic Soft Logic [1] integrate First Order Logic (FOL) and graphical models. More recently, many attempts have been focusing on integrating reasoning with uncertainty with deep learning [20]. A common solution, followed by approaches like Semantic Based Regularization [4] and Logic Tensor Networks [5], relies on using deep networks to approximate the FOL predicates, and the overall architecture is optimized end-to-end by relaxing the FOL into a differentiable form, which translates into a set of constraints. For the sake of overall consistency, one question that can naturally arise in this context is how the fitting of the supervised examples can be expressed using logic formalism. Following this starting point, this paper follows an orthogonal approach for the definition of a loss function, by studying the relation between the translation of the prior knowledge using t-norms and the resulting loss function. In particular, the notion of t-norm generator plays a fundamental role in the behavior of the corresponding loss. Remarkably, the cross-entropy loss can be naturally derived within this framework. However, the presented theoretical results suggest that there is a larger class of loss functions that correspond to the different possible translations of logic using t-norms, and some loss functions are potentially more effective than the cross-entropy to limit the vanishing gradient issue, therefore proving a faster convergence rate.

The paper is organized as follows: Section 2 presents the basic concepts about t-norms, generators and aggregator functions. Section 3 introduces the learning frameworks used to represent supervised learning in terms of logic rules, while Section 4 presents the experimental results and, finally, Section 5 draws some conclusions.

2 Fuzzy Aggregation Functions

The aggregation takes place on a set of values typically representing preferences or satisfaction degrees restricted to the unit interval $[0,1]$ to be aggregated. There are several ways to aggregate them into a single value expressing an overall combined score, according to what is expected from such mappings. The purpose of aggregation functions is to combine inputs that are typically interpreted as degrees of membership in fuzzy sets, degrees of preference or strength of evidence. Aggregation functions have been studied by several authors in the literature [2, 3], and they are successfully used in many practical applications, for instance see [8, 19]. Please note that the fuzzy aggregation functions that will be covered in this section can be directly applied to the output of a multi-task classifier, when implemented via a neural network with sigmoidal or softmax output units.

2.0.1 Basic Definitions.

Aggregation functions are defined for inputs of any cardinality, however for simplicity the main definitions are provided only for the binary case. A (binary) aggregation function is a non-decreasing function $A:[0,1]^{2}\rightarrow[0,1]$, such that: $A(0,0)=0$, $A(1,1)=1$. An aggregation function $A$ can be categorized according to the pointwise order in Equation 1 as: conjunctive when $A\leq\min$, disjunctive when $\max\leq A$, averaging (a mean) when $\min<A<\max$ and hybrid otherwise; where $\min$ and $\max$ are the aggregation functions for the $minimum$ and $maximum$ respectively.

[TABLE]

Conjunctive and disjunctive type functions combine values as if they were related by a logical AND and OR operations, respectively. On the other hand, averaging type functions have the property that low values can be compensated by high values. Mean computation is the most common way to combine the inputs, since it assumed the total score cannot be above or below any of the inputs, but it depends on all the inputs.

2.1 Archimedean T-Norms

Despite averaging functions have nice properties to aggregate fuzzy values, they are not suitable to represent neither a conjunction nor a disjunction, because they do not generalize their boolean counterpart. This is a reason why, we focus on t-norms and t-conorms  [11, 14], that are associative, commutative aggregation functions with 1 and 0 as neutral element, respectively. Table 1 reports Gödel, Lukasiewicz and Product t-norms, which are referred as the fundamental t-norms because all the continuous t-norms can be obtained as ordinal sums of the two fundamental t-norms [10]. A simple example of a t-norm that is not continuous is given by the Drastic t-norm $T_{D}$, that is always returning a zero value, except for $T_{D}(1,1)=1$. Archimedean t-norms [13] are a class of t-norms that can be constructed by means of unary monotone functions, called generators.

Definition 1

A t-norm $T$ is said to be Archimedean if for every $x\in(0,1)$, $T(x,x)<x$. In addition, $T$ is said strict if for all $x\in(0,1)$, $0<T(x,x)<x$ otherwise is said nilpotent.

For instance, the Lukasiewicz t-norm $T_{L}$ is nilpotent, the Product t-norm $T_{\Pi}$ is strict, while the Gödel one $T_{M}$ is not archimedean, indeed $T_{M}(x,x)=x$, for all $x\in[0,1]$. The Lukasiewicz and Product t-norms are enough to represent the whole classes of nilpotent and strict Archimedean t-norms [14].

A fundamental result for the construction of t-norms by additive generators is based on the following theorem [12]:

Theorem 2.1

Let $g:[0,1]\to[0,+\infty]$ be a strictly decreasing function with $g(1)=0$ and $g(x)+g(y)\in Range(g)\cup[g(0^{+}),+\infty]$ for all $x,y$ in $[0,1]$, and $g^{(-1)}$ its pseudo-inverse. Then the function $T:[0,1]\to[0,1]$ defined as

[TABLE]

is a t-norm and $g$ is said an additive generator for $T$.

Any t-norm $T$ with an additive generator $g$ is Archimedean, if $g$ is continuous then $T$ is continuous, $T$ is strict if and only if $g(0)=+\infty$, otherwise it is nilpotent.

Example 1

If we take $g(x)=1-x$, then also $g^{-1}(y)=1-y$ and we get $T_{L}$:

[TABLE]

Example 2

Taking $g(x)=-\log(x)$, we have $g^{-1}(y)=e^{-y}$ and we get $T_{\Pi}$:

[TABLE]

Eq. (2) allows to derive the other fuzzy connectives as function of the generator:

[TABLE]

If $g$ is expressed as a parametric function, it is possible to define families of t-norms, which can be constructed by the generator obtained when setting the parameters to specific values. Several parametric families of t-norms have been introduced [2]. The experimental section of this paper employs the family of Schweizer–Sklar and Frank t-norms, depending on a parameter $\lambda\in(-\infty,+\infty)$ and $\lambda\in[0,+\infty]$ respectively, and whose generators are defined as:

[TABLE]

3 From Formulas to Loss Functions

A learning process can be thought of as a constraint satisfaction problem, where the constraints represent the knowledge about the functions to be learned. In particular, multi-task learning can be expressed via a set of constraints expressing the fitting of the supervised examples, plus any additional abstract knowledge.

Let us consider a set of unknown task functions ${\bf P}=\{p_{1},\ldots,p_{J}\}$ defined on $\mathbb{R}^{n}$, all collected in the vector $\boldsymbol{p}=(p_{1},\ldots,p_{J})$ and a set of known functions or predicates $\boldsymbol{S}$. Given the set $\mathcal{X}\subseteq\mathbb{R}^{n}$ of available data, a learning problem can be generally formulated as $\min_{\boldsymbol{p}}\mathcal{L}(\mathcal{X},\boldsymbol{S},\boldsymbol{p})$ where $\mathcal{L}$ is a positive-valued functional denoting a certain loss function. Each predicate is approximated by a neural network providing an output value in $[0,1]$. The available knowledge about the task functions consists in a set of FOL formulas $KB=\{\varphi_{1},\ldots,\varphi_{H}\}$ and the learning process aims at finding a good approximation of each unknown element, so that the estimated values will satisfy the formulas for the input samples. Since any formula is true if it evaluates to 1, in order to satisfy the constraints we may minimize the following loss function:

[TABLE]

where any $\lambda_{h}$ is the weight for the $h$-th logical constraint, which can be selected via cross-validation or jointly learned [15, 21], $f_{h}$ is the truth-function corresponding to the formula $\varphi_{h}$ according to a certain t-norm fuzzy logic and $L$ is a decreasing function denoting the penalty associated to the distance from satisfaction of formulas, so that $L(1)=0$. In the following, we will study different forms for the $L$ cost function and how it depends on the choice of the t-norm generator. In particular, a t-norm fuzzy logic generalizes Boolean logic to variables assuming values in $[0,1]$ and is defined by its t-norm modeling the logical AND [9]. The connectives can be treated using the fuzzy generalization of first–order logic that was first proposed by Novak [17]. The universal and existential quantifiers occurring in the formulas in $KB$ allows the aggregation of different evaluations (groundings) of the formulas on the available data. For instance, given a formula $\varphi(x_{i})$ depending on a certain variable $x_{i}\in\mathcal{X}_{i}$, where $\mathcal{X}_{i}$ denotes the available samples for the $i$-th argument of one of the involved predicates in $\varphi$, we may convert the quantifiers as the minimum and maximum operations that are common to any t-norm fuzzy logic:

[TABLE]

3.1 Loss Functions by T-Norms Generators

A quantifier can be seen as a way to aggregate all the possible groundings of a predicate variable that, in turn, are $[0,1]$-values. Different aggregation functions have also been considered, for example in [5], the authors consider a mean operator to convert the universal quantifier. However this has the drawback that also the existential quantifier has the same semantics conversion and then it is determined by the authors via Skolemization. Even if this choice may yield some learning benefits, it has no direct justification inside a logic theory. Moreover it does not suggest how to map the functional translation of the formula into a constraint. In the following, we investigate the mapping of formulas into constraints by means of generated t-norm fuzzy logics, and we exploited the same additive generator of the t-norm to map the formula into the functional constraints to be minimized, i.e. $L=g$.

Given a certain formula $\varphi(x)$ depending on a variable $x$ that ranges in the set $\mathcal{X}$ and its corresponding functional representation $f_{\varphi}(x,\boldsymbol{p})$ evaluated on each $x\in\mathcal{X}$, the conversion of universal and existential quantifiers should have semantics equivalent to the AND and OR of the evaluation of the formula over the groundings, respectively. This can be realized by directly applying the t-norm or t-conorms over the groundings. For instance, for the universal quantifier:

[TABLE]

where $g$ is an additive generator of the t-norm $T$ corresponding to the universal quantifier. Since any generator function is decreasing, in order to maximize the satisfaction of $\forall x\,\varphi(x)$ we can minimize $g$ applied to Equation 6, namely:

[TABLE]

As a consequence, with respect to the convexity of the expressions in Equations 7-8, we get the following result, that is an immediate consequence of how the convexity is preserved by function composition.

Proposition 1

If $g$ is a linear function and $f_{\varphi}$ is concave, Equation 7 is convex. If $g$ is a convex function and $f_{\varphi}$ is linear, Equation 8 is convex.

Example 3

If $g(x)=1-x$ (Lukasiewicz t-norm) from Equation 7 we get:

[TABLE]

Hence, in case $f_{\varphi}$ is concave (see [6] for a characterization of the concave fragment of Lukasiewicz logic), this function is convex.

If $g=-\log$ (Product t-norm) from Equation 8 we get the cross-entropy:

[TABLE]

As we already pointed out in Section 2, if $g$ is an additive generator for a t-norm $T$, then the residual implication and the biresidum with respect to $T$ are given by Equation 3. In particular, if $p_{1},p_{2}$ are two unary predicates functions sharing the same input domain $\mathcal{X}$, and $\boldsymbol{S}=\emptyset$ the following formulas yield the following penalty terms:

[TABLE]

3.2 Redefinition of supervised Learning with Logic

In this section, we study the case of supervised learning w.r.t. the choice of a certain additive generator. Let us consider a multi-task classification problem with predicates $p_{j},j=1,\dots,J$ defined over the same input domain with a supervised training set $\mathcal{T}=\{(x_{i},y_{i})\}$ where each $y_{i}\in\{1,2,\ldots,J\}$ is the output class for the pattern $x_{i}$ and $\mathcal{X}$ is the overall set of supervised patterns. Finally, the known predicate $S_{j}$ is defined for each predicate such that $S_{j}(x_{i})=1$ iff $y_{i}=j$, and we indicate as $\mathcal{X}_{j}=\{x_{i}\in\mathcal{X}:S_{j}(x_{i})=1\}$ the set of positive examples for the $j$-th predicate. Then, we can enforce the supervision constraints for $p_{j}$ as:

[TABLE]

In the special case of the predicates implemented by neural networks and exclusive multi-task classification, where each pattern should be assigned to one and only one class, the exclusivity can be enforced using a softmax output activation. Typically, in this scenario, only the positive supervisions are explicitly listed, and since it holds that $g(S_{j}(x))=0,\forall x\in\mathcal{X}_{j}$, yields:

[TABLE]

For instance, in the case of Lukasiewicz and Product logic, we have, respectively:

[TABLE]

corresponding to the $L_{1}$ and cross entropy losses, respectively.

4 Experimental Results

The proposed framework allows to recover well-known loss functions by expressing the fitting of the supervision using logic and then carefully selecting the t-norm used to translate the resulting formulas. However, a main strength of the proposed theory is that it becomes possible to derive new principled losses starting from any family of parametric t-norms. Driven by the huge impact that cross-entropy gained w.r.t. to classical loss functions in improving convergence speed and generalization capabilities, we designed a set of experiments to investigate how the choice of a t-norm can lead to a loss function with better performances than the cross-entropy loss. The Schweizer–Sklar and the Frank parametric t-norms defined in Section 2.1 have been selected for this experimental evaluation, given the large spectrum of t-norms that can be generated by varying their $\lambda$ parameter. The well known MNIST dataset is used as benchmark for all the presented experiments. In order to have a fair comparison, the same neural network architecture is used during all the runs: a 1-hidden layer neural network with 50 hidden ReLU units and 10 softmax output units. The softmax activation function allows to express only positive supervisions, like commonly done in mutually exclusive classification using the cross-entropy loss. Optimization is carried on using Vanilla gradient descent with a fixed learning rate of $0.01$.

Results are shown in Figure 1, that reports the accuracy on the test set of a neural network trained on the MNIST dataset. Specific choices of the parameter $\lambda$ recover classical loss functions, like the cross-entropy loss, which is equivalent to the loss obtained using $T_{\Pi}$. The results confirm that the cross-entropy loss converges faster than the $L_{1}$ obtained when using $T_{L}$. However, there is a wide range of possible choices for the parameter $\lambda$ that brings an even faster convergence and better generalization than the widely adopted used cross-entropy.

5 Conclusions

This paper presents a framework to embed prior knowledge expressed as logic statements into a learning task, showning how the choice of the t-norm used to convert the logic into a differentiable form defines the resulting loss function used during learning. When restricting the attention to supervised learning, the framework recovers popular loss functions like the cross-entropy loss, and allows to define new loss functions corresponding to the choice of the parameters of t-norm parametric forms. The experimental results show that some newly defined losses provide a faster convergence rate that the commonly used cross-entropy loss. Future work will focus on testing the loss functions in more structured learning tasks, like the one commonly addressed with Logic Tensor Networks and Semantic based Regularization. The parametric form of the loss functions allows to define joint learning tasks, where the loss parameters are co-optimized during learning, for example using maximum likelihood estimators.