Iterative Optimization of Multidimensional Functions on Turing Machines under Performance Guarantees

TL;DR

This paper explores the limitations of iterative block Gauss--Seidel algorithms in convex optimization, showing that effective termination with guaranteed accuracy is generally impossible due to computational and reachability constraints.

Contribution

It demonstrates fundamental limitations in algorithmically achieving guaranteed convergence for certain convex minimization problems using block Gauss--Seidel methods.

Findings

Effective termination with performance guarantees is generally impossible.

Some iteration steps cannot be effectively implemented on digital computers.

Not all computable minimizers are reachable by the Gauss--Seidel method.

Abstract

This paper studies the effective convergence of iterative methods for solving convex minimization problems using block Gauss--Seidel algorithms. It investigates whether it is always possible to algorithmically terminate the iteration in such a way that the outcome of the iterative algorithm satisfies any predefined error bound. It is shown that the answer is generally negative. Specifically, it is shown that even if a computable continuous function which is convex in each variable possesses computable minimizers, a block Gauss--Seidel iterative method might not be able to effectively compute any of these minimizers. This means that it is impossible to algorithmically terminate the iteration such that a given performance guarantee is satisfied. The paper discusses two reasons for this behavior. First, it might happen that certain steps in the Gauss--Seidel iteration cannot be effectively implemented on a digital computer. Second, all computable minimizers of the problem may not be reachable by the Gauss--Seidel method. Simple and concrete examples for both behaviors are provided.