Incorporating Shape Knowledge into Regression Models

TL;DR

This paper introduces SIASCOR, a semi-infinite programming approach for incorporating shape knowledge like monotonicity and convexity into regression models, demonstrating its effectiveness on manufacturing and artificial data.

Contribution

It proposes a novel semi-infinite programming method for shape-constrained regression, including an adaptive feasible-point algorithm ensuring optimality and strict constraint satisfaction.

Findings

SIASCOR outperforms data-driven AutoML in shape-constrained settings.

The method successfully applies to manufacturing and artificial examples.

A new methodology (ISI) captures shape knowledge for previously unconsidered applications.

Abstract

Informed learning is an emerging field in machine learning that aims to compensate for insufficient data with prior knowledge. Shape knowledge covers many types of prior knowledge concerning the relationship of a function's output with respect to input variables, for example, monotonicity, convexity, etc. This shape knowledge -- when formalized into algebraic inequalities (shape constraints) -- can then be incorporated into the training of regression models via a constraint problem formulation. The defined shape-constrained regression problem is, mathematically speaking, a semi-infinite program (SIP). Although off-the-shelf algorithms can be used at this point to solve the SIP, we recommend an adaptive feasible-point algorithm that guarantees optimality up to arbitrary precision and strict fulfillment of the shape constraints. We apply this semi-infinite approach for shape-constrained regression (SIASCOR) to three application examples from manufacturing and one artificial example. One application example has not been considered in a shape-constrained regression setting before, so we used a methodology (ISI) to capture the shape knowledge and define corresponding shape constraints. Finally, we compare the SIASCOR method with a purely data-driven automated machine learning method (AutoML) and another approach for shape-constrained regression (SIAMOR) that uses a different solution algorithm.