Methodology

The perturbation paradigm.

We measure reasoning by flip rate — a paired-item test that precludes the surface-feature heuristics most models use to game conventional evaluations. Each test consists of two items: one canonical, one perturbed. The pair is the unit of measurement, not the item. A model that gets one item right and the other wrong has not reasoned; it has matched. Flip rate measures how often that happens.

Conventional accuracy rewards surface-feature heuristics. Flip rate punishes them. The result is a far harder, far more honest signal — and one that any lab can apply to its own model.

Pairs above are verbatim records from the ConvergeMini rail of the eval kit. The slider scales two axes together: hop count (the number of inference steps from premise to answer) and distractor density (the number of facts irrelevant to the conclusion). The single fact flipped between canonical and perturbed — marked with an indigo bar — is the load-bearing change. The correct answer flips with it; a reasoner tracks the change through the chain, a surface-matcher does not.