We Put Our Rankings Up Against an Outside Model. It Was a Tie.

Why we tested it

A football ranking is easy to publish and hard to check. A board only means something if its forecasts hold up on games it has not seen. So we put an independent ranking framework, built on a published method from outside our team, up against the Newton rating model. Both were scored on the same 1,597 FBS-vs-FBS games from 2024 and 2025, regular season and postseason, and both received the same information at the same Monday cutoffs each week.

Before we trusted the comparison, we rebuilt the outside method from its published description and checked that our version reproduced its published forecasts on all 4,478 test games, to floating-point precision. A head-to-head only means something if both sides are really doing what they claim, so that check came first.

The result: a tie with a small lean

We scored each model by log loss, which rewards forecasts that gave the actual result a high probability. Lower is better. Newton finished at 0.5356 per game and the outside framework at 0.5399. The gap is 0.0043 per game, with a 95% interval running from slightly below zero to +0.0131 when we resample by week. That interval crosses zero, so we call it a tie.

Newton was ahead in both seasons, by 0.0064 in 2024 and 0.0023 in 2025. The outside model came out ahead in the postseason (92 games) and in top-25 matchups (120 games), but neither subset is large enough to say anything clear.

One caveat belongs right next to the result. Our recipes were developed while 2024 and 2025 were visible to us, so this is a replay, not a set of blind forecasts. That can tilt the result toward the model we built, and it is the main reason we say tie and not win.

What we learned

Two lessons came out of the comparison, and both changed how we build.

The first is about the start of the season. In the main test, both models carried last season forward. When we turned that off, our models lost about 0.016 log loss per game across the whole test: about 0.035 in Weeks 0-4 and about 0.010 after that. Last season matters most when this season has few games to learn from.

That raised a real choice. On October 18 we decided that team ranks use this season only, so the board reflects what has happened on the field this year. Under that rule, our this-season rank model clearly beat the outside model's this-season version on the same games, 0.5515 to 0.5701 per game. Almost all of that edge came in Weeks 0-4. The reason is the starting point: our model begins each team from a preseason estimate of roster talent, while the outside version starts every team from the same blank slate. From Week 5 on, the two were close.

The rule still has a measured cost. In a separate rankings study on 2024 and 2025 (3,376 forecasts), the new this-season rank model was a little less accurate than the previous Newton rating it replaces on the board: a Brier score of 0.1900 against 0.1855, a margin error of 12.42 points against 12.28, and 70.6% of winners against 70.9%. It is still more accurate than the older board order we used before that. We kept the rule and report the cost openly. Newton's picks are unchanged by it.

The second lesson is about uncertainty. A ranking that says "Team A is 12th" is only part of the story. The more honest version says where Team A is likely to finish, as a range. In that same rankings study, the this-season model's calibrated 80% rank range covered 79.7% of season-end ranks, with an average width of 30.8 spots. The previous Newton rating's range covered 82.5% at a width of 24.8. The oldest ranges covered 93.9%, but they were 42.8 spots wide, which is too wide to tell a fan much. A range that holds up without being vague is the goal. These figures come from 2024 and 2025, seasons we had already studied, so treat them as a check on calibration, not as a blind result.

Two numbers fans now see

The framework gave us two ideas, and we built both on our own model rather than adopting the outside system.

The first is a team's chance to beat an average FBS team, shown to Premium accounts. The second is a team's chance to finish the regular season ranked No. 1, shown on every account. Both come from the this-season rank model. Both rest on the same point: a single rank hides how much a team's position can move, and a spread of outcomes says more.

Where the betting market still leads

The betting market is still the sharpest benchmark we have. In the same comparison, on 1,505 regular-season games where a consensus closing line was available, the closing line was more accurate than either model, on both log loss and winners picked. The gap between Newton and the line is not statistically clear at this sample size. The gap between the outside model and the line is. Lines are referenced here for comparison only. Not betting advice.

What comes next

At the end of 2025, our board and the outside board agreed closely, with a rank correlation of 0.978 and 23 of the top 25 teams in common. Our board has Ohio State first and Indiana second, and the few differences between the two boards include the very top. Neither board has been checked against an outside truth, and we will not claim one is right until we have that check.

The next step is to keep score on forecasts made before the games are played. The tie is a useful answer. It tells us the model is competitive, and it tells us exactly where we still need to prove more.