Tuesday, August 19, 2014

Methods and scores used for verifying ensemble forecasts

http://www.cawcr.gov.au/projects/EPSverif/scores/scores.html

http://www.eumetcal.org/resources/ukmeteocal/temp/msgcal/www/english/msg/ver_categ_forec/uos2/uos2_ko4.htm




FCST yes FCST no  Total
OBS yes hits misses observed yes
OBS no false alarms correct negatives observed no
Total forecast yes forecast no total



Root mean square error

Equation for root mean square error
Answers the question: What is the magnitude of the forecast errors?
Range: 0 to infinity.  Perfect score: 0.


Equitable threat score


Equation for equitable threat score
where Equation for hits due to random chance
Answers the question: How well did the forecast occurrence of events correspond to the actual (observed) occurrence of events?
Range: -1/3 to 1, 0 indicates no skill.   Perfect score: 1.


Bias score

Equation for bias
Answers the question: How does the forecast frequency of events compare to the actual (observed) frequency of events?
Range: 0 to infinity.  Perfect score: 1.


Brier score
Expansion of Brier Score
Answers the question: What is the magnitude of the probability forecast errors?
Range: 0 to 1.  Perfect score: 0.


Brier skill score
Equation for Brier skill score
Answers the question: What is the relative skill of the probabilistic forecast over that of climatology, in terms of predicting whether or not an event occurred?
Range: minus infinity to 1, 0 indicates no skill when compared to the reference forecast. Perfect score: 1.


Ranked probability score
ranked probability score formula
Answers the question: How well did the probability forecast predict the category that the observations fell into?
Range: 0 to 1.  Perfect score: 0.


Ranked probability skill score 
Equation for ranked probability skill score
Answers the question: What is the relative skill of the probabilistic forecast over that of climatology, in terms of getting close to the actual outcome?
Range: minus infinity to 1, 0 indicates no skill when compared to the reference forecast. Perfect score: 1.


 Relative value (value score)
Value score
Answers the question: For a cost/loss ratio C/L for taking action based on a forecast, what is the relative improvement in economic value between climatalogical and perfect information?
Range: minus infinity to 1.  Perfect score: 1.

No comments:

Post a Comment