Politics

Sometimes, there are more sophisticated-seeming criticisms. “Sure, your forecasts are probabilistic,” people who think they’re very clever will say. “But all that means is that you can never be wrong. Even a 1 percent chance happens sometimes, after all. So what’s the point of it all?”

I don’t want to make it sound like we’ve had a rough go of things overall.1 But we do think it’s important that our forecasts are successful on their own terms — that is, in the way that we have always said they should be judged. That’s what our latest project — “How Good Are FiveThirtyEight Forecasts?” — is all about.

That way is principally via calibration. Calibration measures whether, over the long run, events occur about as often as you say they’re going to occur. For instance, of all the events that you forecast as having an 80 percent chance of happening, they should indeed occur about 80 out of 100 times; that’s good calibration. If these events happen only 60 out of 100 times, you have problems — your forecasts aren’t well-calibrated and are overconfident. But it’s just as bad if they occur 98 out of 100 times, in which case your forecasts are underconfident.