Once the criteria are written, two things decide whether they matter: how the bar was set, and how many times you let yourself look. The second is the counter-intuitive one — repeated checking manufactures significance by itself, even if you never change a number.
| Gate | Blocks |
|---|---|
| ① Magnitude: gross excess ≥ cost line | "there is an edge but it does not pay costs" |
| ② Significance: z ≥ 2.0 | "big enough, but it is luck" |
| ③ Beat the nulls: all three | "it is actually the market, not the strategy" |
| ④ Stability: halves agree in sign | "one stretch is carrying everything" |
Each blocks a different false positive and none substitutes for another. My own final read is the cleanest illustration: ③ and ④ PASS, ① and ② FAIL.
"It beat three null models and both halves agree" sounds like something real. But the magnitude was +1.56pp against a 2.25pp cost line.
Beating random betting is not beating costs. A strategy at +1.56pp of excess with 2.25pp of cost is not "slightly profitable" — it is losing with certainty on every single trade.
"It beat the null" is the sentence people use to convince themselves. It establishes only that the strategy is not pure noise; it says nothing about covering the fee.
The data is sitting there. Why not look more often?
| Practice | What it actually is |
|---|---|
| Accumulate to n, read once | one test |
| Check daily, stop when good | many tests, reporting only the best |
| Not good? "give it more time" | the same, and it can never fail |
That last row is the crux: an experiment that permits "give it more time" has no failure state. It has only "succeeded" and "not yet" — which is not an experiment, it is waiting.
| Allowed | Forbidden |
|---|---|
| watch the current number | change criteria because of it |
| watch trends, check data quality | stop early or extend because of it |
| stop for instrument failure | stop because the number is ugly |
The third row's distinction matters: stopping because the instrument is broken and stopping because the result is bad are different acts. The first is mandatory — the data is dirty and continuing is pointless. The second is cheating. The test: "if the number were good, would I still stop for this reason?"
One case: when the remaining sample cannot mathematically change the outcome.
My final read worked exactly that way. Trigger was n=2,500; the actual read was n=2,468 (98.7%). The justification recorded in the verdict:
An earlier kill used the same arithmetic: at n≈460 with excess −1.8pp against a +4pp gate, the remaining ~40 windows would each need +71pp, above the single-window ceiling.
It may only be used to kill early, never to declare success early.
Because "can no longer pass" is a strictly computable fact, while "is now certain to pass" never is — later samples can always deteriorate.
The asymmetry is deliberate: killing early saves continued burn, declaring early saves testing you should still be doing. One is a gain; the other is a risk.
Threshold and sample size are set together, never apart:
In practice you invert it: the size of edge you want to detect determines the sample you need. Mine estimated a standard error of about 1.0pp at n≈2,500, so z ≥ 2.0 corresponds roughly to "excess of at least 2pp". That lines up with the 2.25pp threshold computed from costs — the two gates cross-check each other rather than being independently invented.
A practical corollaryif the sample you need far exceeds what you can ever collect, the line should be abandoned before it starts — you will never obtain data capable of a conclusion. That judgement is free, and almost nobody makes it.
Gates must be computed, run in parallel, and frozen in advance; the data is read once. An experiment that permits "give it more time" has no failure state — and something that cannot fail cannot demonstrate success either.