Every backtest is an approximation — that is not the problem. The problem is not knowing which way it errs. A number labelled "optimistic upper bound" is still usable; an unlabelled number cannot support any decision, because you cannot tell "slightly short" from "nowhere near".
| Approximation | Direction | Why |
|---|---|---|
| Fill at the best price | optimistic | live orders walk the book |
| Fill at the midpoint | more optimistic | the spread is not counted at all |
| Assume resting orders fill | optimistic | filling is not random |
| Drop unfilled orders from the sample | severely optimistic | that is selection on the outcome |
| Conservative queue assumptions | pessimistic | less benign flow, more toxic flow |
| Simulate a short tenor from long-tenor flow | optimistic | short-tenor flow is sparser |
| Ignore price drift during execution | optimistic | drift is large on short tenors |
Note the distribution: six of seven lean optimistic. Not a coincidence — each simplification exists to make the backtest runnable, and simplifications almost always remove a cost, rarely a revenue.
I needed to estimate how much of a maker's inventory would pair into sets. Version one used truncated replay: take a long-tenor wallet's real flow, pretend it were a short holding window, recompute.
Later I replayed against recorded real books plus complete tape:
| Tenor | Truncated replay | Full simulation | Overstated |
|---|---|---|---|
| 5 minutes | 47.0% | 36.3% | 1.3× |
| 15 minutes | 70.9% | 33.6% | 2.1× |
| 1 hour | 95.4% | 54.9% | 1.7× |
It had been "undecided" — 70.9% looked high enough to be worth trying. The full simulation returned 33.6%, worse than five minutes. Killed outright.
The failure was not inaccuracy. It was that I had not judged in advance which way it would err. Thinking through "does simulating a short tenor from long-tenor flow overstate or understate pairing" would have marked 70.9% an optimistic upper bound — unusable for a decision.
Once you know the direction, something useful becomes available: quote both bounds. For one path I had two numbers:
| Source | Result | Direction |
|---|---|---|
| Full simulation, conservative queue | −3¢/share | pessimistic lower bound |
| A skilled operator's measured result | +0.1¢/share | optimistic upper bound |
Namely: is there evidence I would land at the good end? There was not, so the line was never run. With one mid-point number that question cannot even be posed — you would think you had an answer when you had an unattributed point estimate.
| Assumption | If wrong | Cost |
|---|---|---|
| Optimistic | backtest positive, live negative | discovered with real money; unbounded |
| Pessimistic | backtest negative, live might have worked | a missed opportunity; bounded |
The two errors have wildly asymmetric costs, so wherever it is genuinely uncertain, resolve it against yourself. That is not pessimism, it is asymmetric risk management.
| Ask | Failing answer |
|---|---|
| How was the fill price determined? | "the close" — no spread, no book-walking |
| How do resting orders fill in this model? | "the framework default" — you do not know the direction |
| Where did unfilled orders go? | "filtered out" — the sample is contaminated |
| Is this an upper or a lower bound? | cannot say → unusable |
| Is there evidence for the good end? | "probably fine" |
| Which ruler judges wins? | not the one the market settles on → see below |
The last cost me the most: the ruler judging wins disagreed with market settlement on 10.1% of all windows, with a direction — enough to manufacture a stable-looking pattern. That is beyond a bias direction; it is the wrong instrument entirely — see the broken ruler.
Every backtest number should carry one sentence: is this an upper bound or a lower bound? Without it the number cannot support a decision, because you cannot distinguish "slightly short" from "nowhere near" — and those differ by an order of magnitude in cost.