Shootout-89: A Comparative Evaluation of Knowledge-based Systems that Forecast Severe Weather

TL;DR

The paper evaluates six AI-based weather forecasting systems during Shootout-89, comparing their accuracy, usability, and skill in predicting severe weather over northeastern Colorado, highlighting traditional expert systems' relative performance.

Contribution

First comprehensive comparison of diverse AI-based weather forecast systems in operational conditions, assessing skill, usability, and knowledge requirements.

Findings

Traditional expert systems best discriminated significant weather events.

All systems had skill comparable to persistence and climatology.

Systems varied in meteorological expertise needed and statistical behavior.

Abstract

During the summer of 1989, the Forecast Systems Laboratory of the National Oceanic and Atmospheric Administration sponsored an evaluation of artificial intelligence-based systems that forecast severe convective storms. The evaluation experiment, called Shootout-89, took place in Boulder, and focussed on storms over the northeastern Colorado foothills and plains (Moninger, et al., 1990). Six systems participated in Shootout-89. These included traditional expert systems, an analogy-based system, and a system developed using methods from the cognitive science/judgment analysis tradition. Each day of the exercise, the systems generated 2 to 9 hour forecasts of the probabilities of occurrence of: non significant weather, significant weather, and severe weather, in each of four regions in northeastern Colorado. A verification coordinator working at the Denver Weather Service Forecast Office gathered ground-truth data from a network of observers. Systems were evaluated on the basis of several measures of forecast skill, and on other metrics such as timeliness, ease of learning, and ease of use. Systems were generally easy to operate, however the various systems required substantially different levels of meteorological expertise on the part of their users--reflecting the various operational environments for which the systems had been designed. Systems varied in their statistical behavior, but on this difficult forecast problem, the systems generally showed a skill approximately equal to that of persistence forecasts and climatological (historical frequency) forecasts. The two systems that appeared best able to discriminate significant from non significant weather events were traditional expert systems. Both of these systems required the operator to make relatively sophisticated meteorological judgments. We are unable, based on only one summer's worth of data, to determine the extent to which the greater skill of the two systems was due to the content of their knowledge bases, or to the subjective judgments of the operator. A follow-on experiment, Shootout-91, is currently being planned. Interested potential participants are encouraged to contact the author at the address above.