Preregistration does one thing: write down what counts as success before you see the result. It sounds like paperwork. It is the only tool that stops one particular class of error — the class whose defining feature is that you will never notice you committed it, because every step looked reasonable.
Not lying. Deceiving yourself honestly. The usual sequence:
| Step | What you thought | What happened |
|---|---|---|
| 1 | "let's just run it and see" | no success criterion exists |
| 2 | "that number is poor, maybe the sample is small" | keep running, wait for a better one |
| 3 | "let me try slicing it differently" | a dozen subsets appear |
| 4 | "this subset looks great" | you picked the largest number in the noise |
| 5 | "found an edge" | you found the selection process |
Every step is intuitive; together they manufacture a false finding. And you will genuinely believe it, because you did not lie at any point.
In one analysis I sliced twelve lines × two halves = 24 subsets and reported the good ones. I wrote a warning into the document at the time: "apart from a single z=3.28, every z falls between 1.2 and 1.9 — this is hypothesis generation, not a result."
Writing that took ten seconds, and it told everyone afterwards — me included — that those numbers could not be traded on. That is the minimum viable version of preregistration: record that you selected.
| Item | Why in advance |
|---|---|
| Which line is judged | stops swapping horses: running five and reporting the best |
| The threshold | stops "just short, but the direction is right" |
| The sample size to read at | stops stopping at the prettiest moment |
| When you read | stops repeated peeking until a good number appears |
| What counts as failure | failure must be as explicit as success |
| The date you wrote it | makes "before or after" a verifiable fact |
The one that killed my line had four thresholds: gross excess ≥ a computed cost line, significance z ≥ 2.0, beat three contemporaneous null models, and both halves of the sample agreeing in sign — all frozen seven days before the read.
"Gross excess ≥ 4%" is a number pulled from the air. A correct threshold is derived from costs:
And write down how the threshold may move, in advance. Mine said: "if measured slippage > 0, raise accordingly." Measured slippage came in at +0.8pp, so the threshold automatically rose from 1.45pp to 2.25pp — a clause firing, not a criterion being revised after the fact.
That 1.45 was "1.75 of fee minus 0.30 of rebate stack". The 0.30pp never arrived — not one cent.
Add it back and the true threshold was 1.75 + 0.8 = 2.55pp. In other words, I spent money I had not received while computing my own bar.
Rulea cost line may contain only items confirmed to occur. This error's direction is always to lower the bar, which is why it will never be caught by its own consequences.
The first paragraph of mine reads, verbatim:
Plainly: the line under judgement was chosen after peeking at almost a quarter of the sample, and partly because it happened to be ahead.
That is a real contamination. It sits in the first paragraph rather than hidden. The reasoning:
| Result | |
|---|---|
| No preregistration | the contamination exists but is invisible; nobody can characterise it later |
| A preregistration pretending to be clean | the contamination is concealed — worse than none |
| One that admits it | the contamination becomes a quantity you can argue about |
A preregistration that admits its contamination still beats not having one — it at least lets a reader ask "how big is this, and could it overturn the conclusion?" And that question cannot be asked at all unless it is written down.
Preregistration earns its keep at exactly one moment: when the data arrives and you dislike it.
My final read: threshold 2.25pp, read +1.56pp; threshold z ≥ 2.0, read z = 1.62 — two fails. But the other two criteria passed (beat three nulls, halves agreed in sign).
Reading only the passes, it is easy to conclude "there is an edge, just not a big one". Criteria 1 and 2 exist to block that conclusion.
① Do not raise the z threshold ("2 is 2") ② Do not reopen a segment ③ Do not delete a single row of data.
The bar was set seven days earlier, so when the number came it was no longer mine to move. A line that needs its bar moved to survive was never alive — moving it only defers the discovery, and it spends real money in the meantime.
No tooling required. Before your next attempt, write four lines in a notes file and save it:
| # | Write |
|---|---|
| 1 | which one I am judging (one line, not a family) |
| 2 | how much sample before I read, and I read once |
| 3 | what number counts as success (computed from costs) |
| 4 | today's date |
Four lines, two minutes. It will not stop every error, but it stops the most expensive class: "in hindsight I think this should count as a success."
Preregistration is not there to prove anything to anyone else. It is there so that the future you cannot lie to the present you. Its only moment of use is when the data lands, you do not like it, and the bar is no longer yours to move.