Optimal Proof Systems for Complex Sets are Hard to Find

TL;DR

This paper constructs oracles demonstrating the inherent difficulty in finding complex sets with optimal proof systems, highlighting longstanding open problems in computational complexity theory.

Contribution

It provides the first oracle-based evidence that certain complex sets lack optimal proof systems, clarifying the difficulty of longstanding open questions.

Findings

No sets in PSPACE extbackslash NP have optimal proof systems relative to O_1.

No sets outside NQP have optimal proof systems relative to O_2.

Existence of arbitrarily complex sets with almost optimal algorithms but no optimal proof systems relative to O_2.

Abstract

We provide the first evidence for the inherent difficulty of finding complex sets with optimal proof systems. For this, we construct oracles $O_1$ and $O_2$ with the following properties, where $\mathrm{RE}$ denotes the class of recursively enumerable sets and $\mathrm{NQP}$ the class of sets accepted in non-deterministic quasi-polynomial time. - $O_1$: No set in $\mathrm{PSPACE} \setminus \mathrm{NP}$ has optimal proof systems and $\mathrm{PH}$ is infinite - $O_2$: No set in $\mathrm{RE} \setminus \mathrm{NQP}$ has optimal proof systems and $\mathrm{NP} \neq \mathrm{coNP}$ Oracle $O_2$ is the first relative to which complex sets with optimal proof systems do not exist. By oracle $O_1$, no relativizable proof can show that there exist sets in $\mathrm{PSPACE} \setminus \mathrm{NP}$ with optimal proof systems, even when assuming an infinite $\mathrm{PH}$. By oracle $O_2$, no relativizable proof can show that there exist sets outside $\mathrm{NQP}$ with optimal proof systems, even when assuming $\mathrm{NP} \neq \mathrm{coNP}$. This explains the difficulty of the following longstanding open questions raised by Kraj\'i\v{c}ek and Pudl\'ak in 1989, Sadowski in 1997, K\"obler and Messner in 1998, and Messner in 2000. - Q1: Are there sets outside $\mathrm{NP}$ with optimal proof systems? - Q2: Are there arbitrarily complex sets outside $\mathrm{NP}$ with optimal proof systems? Moreover, relative to $O_2$, there exist arbitrarily complex sets $L \notin \mathrm{NQP}$ having almost optimal algorithms, but none of them has optimal proof systems. This explains the difficulty of Messner's approach to translate almost optimal algorithms into optimal proof systems.