<ul><li>Forecasting is a difficult task to evaluate as the ground truth is only known in the future.</li><li>A new consistency check framework is proposed to benchmark and evaluate forecasters instantaneously.</li><li>A general consistency metric based on arbitrage is introduced to measure the consistency of predictions.</li><li>An automated evaluation system is built to measure consistency and correlate it with ground truth Brier scores.</li></ul>

Consistency Checks for Language Model Forecasters

Discover more