How SignalXI measures results
Evaluating a model takes more than counting wins. SignalXI records each published recommendation so performance, calibration and stability can be reviewed over time.
1. Record before the outcome
A recommendation should be recorded before the match is settled. We keep the market, selection, odds, probability, classification and date. This reduces the risk of retrospectively showing only favorable outcomes.
2. Hit rate
Hit rate is the percentage of settled recommendations that win. It is intuitive, but incomplete by itself because a high hit rate can still perform poorly when prices are too short.
3. ROI and profit
ROI relates accumulated profit or loss to the units risked. Profit shows the net result in units. These metrics incorporate price and make it easier to compare segments with different average odds.
4. Average odds and distribution
Average odds help place hit rate in context. We also review how recommendations are distributed by market, competition and confidence level so an overall number does not hide important segment differences.
5. Brier Score
Brier Score evaluates the quality of a probability rather than only whether a pick won or lost. It penalizes overconfident estimates when events fail to occur and helps assess calibration.
6. Calibration
A well-calibrated model should, over sufficiently large samples, observe real frequencies close to its estimated probabilities. Events estimated around 70% should resolve positively roughly in that range over time, not necessarily in a small run.
7. Sample size
Short streaks are not treated as conclusive. A handful of wins or losses can move early metrics dramatically, so SignalXI avoids presenting small samples as definitive proof that a model works or has failed.
8. Losses remain visible
Transparency requires keeping negative outcomes. The public results record includes wins and losses because the goal is to evaluate the real system rather than build a favorable retrospective selection.
View tracked results →