How Do You Actually Measure a Rare Event?
Once I stopped trusting that 92 percent, I needed a better way to grade the model. Here’s the thing about grading a rare event. It’s like grading a fire alarm system based on how many days it correctly stayed silent. Of course it’s silent most days, there’s no fire most days. What you actually care about is whether it goes off when there’s an actual fire, and whether it also keeps going off on days when there’s no fire at all.
That’s basically the two questions I needed answered. Did it catch the real events? And did it raise a false alarm?
For the first question, I used a score called Extreme Dependency Score, or EDS. Without getting into the math, think of it as a score that specifically checks how good the model is at catching the rare event, while ignoring all “nothing happened” days that make accuracy look inflated. The model scored strongly on EDS, 0.84 out of 1.
But EDS has a blind spot. It doesn’t care about false alarms.
So I paired it with a second number called frequency bias. It just asks, did the model predict roughly as many storms as actually happened? If it predicted way more storms than occurred, that’s overpredicting. Way fewer, that’s underpredicting. Finally, a perfect score here is 1.
My model landed at 0.90, just slightly under that perfect mark. Put that together with the EDS of 0.84, and now I had a real trustworthy picture. A model that catches most of the real storms.
Comparison among other models
A good score doesn’t mean much floating on its own. So I built two simpler models to measure against my CNN.
The first one was a random prediction model, weighted by how often Kona lows tend to happen during the wet season. It performed about as well as you’d expect from a coin flip. Every score was weak, except frequency bias, which accidentally looked fine simply because random guessing balances itself out over enough tries. It’s a good reminder of why judging a model on just one number can be misleading.
The second was a simple Artificial Neural Network that looked at two summary numbers (PC1 and PC2) describing the wind patterns over the whole North Pacific. Surprisingly, this one wasn’t terrible. Its EDS score came close to my CNN’s. But its frequency bias was predicting storms nearly 9 times more often than they actually happened.
My CNN was the only one that got both numbers right at the same time. That balance is really the whole point here.
The Explainable AI approach: Randomized Input Sampling for Explanation (RISE)
Good scores are one thing. But I wanted to answer a more basic question. Was the model actually learning something real about how these storms form, or did it just get lucky finding some random shortcut in the data?
To check this, I used a technique called RISE. Imagine covering up different parts of a photo , one section at a time, and watching how confident someone stays when they try to guess what’s in the photo. Cover up a part that doesn’t matter, and their confidence barely changes. Cover up the one part they were actually using to make the guess, and their confidence drops fast. Do this over and over across the whole image, and you can map out exactly which parts of the picture actually mattered.
That’s what RISE does to the model. I ran it across every case where the model correctly caught a Kona low, then combined the results into one map. The model marked the important area as the ocean southwest of Hawaii.
That isn’t a random result. Meteorologists have known for decades, through old fashioned manual observation, that Kona lows tend to form around that exact region. The model never read any of that research. Nobody told it where to look. It figured out the answer entirely on its own, just from learning spatial patterns in the wind data.
How did CNN arrive at this conclusion ? We might never know ! Neural networks are sort of a black box. However, that’s a topic for another day.
If you want a deep dive of my thesis, check out my research at State University of New York (UAlbany) : scholarsarchive.library.albany.edu/etd/107