Ranking checkpoints that are too close to call

I’m building a bot that plays Pokémon VGC doubles on Pokémon Showdown. Its action model looks at the battle and picks a move and a target for each of its two Pokémon. I retrain it often, so I keep needing an answer to one question: does this new checkpoint play better than the one I’m serving?

For checkpoints that differ a lot, that’s easy. For siblings, models trained from the same starting point on the same games with one change between them, I haven’t found anything that answers it. What I did find is a test that tells me what a checkpoint gets wrong, which turned out to be more useful.

The checkpoint names below (venusaur, raticate and so on) are internal run labels. They have nothing to do with the Pokémon on the field.

Accuracy doesn’t see blunders

The standard offline score is how often the model’s first choice for its first Pokémon matches what a human did, on held-out replays. It’s cheap and stable, and between siblings it barely moves. My promotion rule treats anything within ±0.3 points as a tie.

The clearest example was a serving-time patch for type immunities. When exactly one foe was a Ghost type, about 1% of one checkpoint’s opening Fake Outs (24 of 2,515) went into the Ghost, which is immune. The patch took that to zero. Accuracy moved by −0.05 points. The patch changed the model’s top choice in only 0.3% of decisions, and in this exact situation the human players in the data made the mistake more often than the model did (3.5%), so agreeing with them was never going to reward the fix.

The arena couldn’t rank siblings

So I built an arena. It runs Showdown’s own battle engine locally, with both sides driven by the same client code the bot uses on the live ladder. I wrote down the statistics and pass bars before playing any games.

It passed the easy checks. In a pre-registered run of 1,800 games, it put a random player and a simple heuristic far below every neural checkpoint. It also found a real targeting bug in my client.

Siblings were another matter. A round robin of five siblings, 2,000 games, separated none of them. Head-to-heads gave confident verdicts that contradicted the offline numbers: one checkpoint that gained 1.22 points of held-out accuracy on choosing both moves together came out worse in the arena, on two replicate runs. And against the ladder, the arena’s sibling order ran backwards: Spearman ρ = −0.6 on one ladder wave of five checkpoints and −0.5 on another of three. One known-weak checkpoint even scored below random, because the arena fed it through an input encoder it was never trained on.

I stopped using the arena to rank checkpoints the same day. It’s now a smoke test for crashes, invalid moves and large regressions.

The ladder can’t referee

The natural conclusion is “the arena is wrong”. I can’t support it, because the ladder is the thing I was checking against, and it’s noisy at this scale.

• At about 100 games per checkpoint, the 95% confidence interval on my ladder performance measure is roughly ±0.09 to ±0.10. That’s about half the spread between the best and worst of the ten siblings below, and wider than most gaps between neighbours.
• A Monte Carlo power analysis found that even a perfect instrument would pass my “ρ ≥ 0.7 across five checkpoints” bar only 20–50% of the time at sibling-scale differences.
• One unpatched checkpoint, laddered in three separate runs, moved by 0.07 on that measure between them. Drift in the opponent pool sits on top of the game-count noise.

So every comparison against the ladder in this post tests two noisy rankings against each other. A failure to correlate doesn’t say which one is wrong.

Boards with proven answers

kcheck asks a narrower question: when a position has a provably right answer, does the checkpoint pick it?

It is a set of doubles positions where the revealed information decides one best action. The proofs use @smogon/calc with this format’s species and move changes. A KO has to hold at my minimum damage against their maximum bulk. A speed order has to hold over every legal stat spread and nature. Each candidate action is classified (KO, immune, hits ally, outsped), and a board is kept only if one class clearly wins. Boards depend on different information, so each one declares what it needs. A board only counts for a checkpoint whose input includes all of it. Near-duplicate boards share a “lesson”, and confidence intervals are bootstrapped over lessons, not boards.

It did separate one pair of siblings. On a 53-board pilot, venusaur-with-patch trailed raticate-with-patch by 6 points, with a confidence interval from −20 to +6 that includes zero. On 1,010 boards the gap was −4.3 points, [−7.1, −1.4]. Audits of the scaled set found a defect rate around 2–3%. A later audit removed boards whose proofs had missed part of the legal nature range, leaving 875. On that frozen set venusaur-with-patch scored 41.4% and raticate-with-patch 44.9%.

It doesn’t separate every pair. The serving checkpoint and raticate-with-patch tie on the frozen set at −1.3 points, [−4.2, +1.7].

What it does every time is name mistakes. On the frozen set, raticate-with-patch got 6 of 45 Trick Room boards right, where a greedy policy that runs the calc on everything visible gets 37. It got 3 of 20 boards where the right move depends on Fairy being immune to Dragon. A multi-turn version runs fixed positions against scripted opponents and flags known-bad moves each turn. It runs on vgc-engine, my Rust simulator, which replayed 98.9% of a sealed sample of real battles identically to Showdown when their random outcomes were forced to match. In an exploratory 50-scenario run, the serving checkpoint made status moves into revealed immunities in 64 of 2,943 decisions: 45 Sleep Powders into grounded Pokémon under Electric Terrain, 16 Thunder Waves into an Electric type and 3 Will-O-Wisps on turn 3 that a Fire-type Volcarona, using Rage Powder every turn, pulled onto itself. Terrain is part of the model’s input, so it could see the problem and hadn’t learned it.

None of that shows up in accuracy, and the arena only reports who won.

It doesn’t rank them either

Before running it, I pre-registered a correlation with the ladder over ten frozen siblings. It wasn’t supported. The primary multi-turn metric gave ρ = −0.47, [−0.86, +0.25]. Single-turn kcheck gave +0.25, [−0.53, +0.87].

A coarser pre-registered check did pass. Three checkpoints already known to be weak on the ladder scored below all ten siblings on both versions; on the easy boards they scored 14–17% against a sibling minimum of 38.5%. That shows coarse validity, where clearly worse models land clearly lower, not a ranking.

What I use now

I’ve stopped expecting any instrument to rank close siblings, because the ladder that would validate the ranking can’t resolve them at the game counts I can afford. Instead:

• The arena is a smoke test.
• kcheck and its multi-turn version, with the accuracy gate, work as a floor: a candidate that gets worse on a class of mistakes doesn’t go to the ladder. They also tell me what to fix.
• The ladder is the final check before a swap.

Settling the ranking question would take far more ladder games per checkpoint than I’ve run.