A real verdict — on my own rules
This is not an example I wrote to look good. It is the actual output of the same gauntlet I offer to run on your rules, applied to mine. 12 of 15 configurations failed. I dropped every one of them.
Why publish this? Because the thing worth checking about a validation service is whether it ever says no. A report that always finds edge is a sales document. These are the numbers my own trading slots produced, and the reason ten of my slots are switched off right now.
The two gates, fixed before the data was seen
A configuration passes only if it clears both:
- PBO < 0.50 — probability of backtest overfitting. Below one half means the best-looking configuration still beats the median out-of-sample more often than not.
- DSR > 0.95 — deflated Sharpe ratio. The Sharpe is still credible after penalising for how many configurations were tried. Try enough combinations and something always looks brilliant; this is the number that charges you for the search.
Both thresholds were fixed in advance, in code, before any of these runs. That is the whole point — a threshold chosen after seeing the result is not a threshold, it is a preference.
The sweep
IS = in-sample, what the backtest showed. OOS = out-of-sample, what happened on data the optimiser never saw.
| slot | OOS trades | OOS win | IS net | OOS net | PBO | DSR | verdict |
|---|---|---|---|---|---|---|---|
| BTC_4h | 109 | 34.9% | +62% | +3.1% | 0.057 | 0.484 | FAIL |
| BTC_1d | 41 | 46.3% | +882% | +210.5% | 0.329 | 0.982 | PASS |
| ETH_4h | 120 | 29.2% | +33% | -11.0% | 0.171 | 0.385 | FAIL |
| ETH_1d | 51 | 31.4% | +1016% | +168.0% | 0.443 | 0.829 | FAIL |
| SOL_4h | 130 | 29.2% | +142% | +6.4% | 0.014 | 0.484 | FAIL |
| SOL_1d | 20 | 55.0% | +1033% | +415.3% | 0.329 | 0.995 | PASS |
| XRP_4h | 107 | 34.6% | +279% | +100.9% | 0.200 | 0.920 | FAIL |
| XRP_1d | 49 | 38.8% | +916% | +251.1% | 0.286 | 0.922 | FAIL |
| BNB_4h | 95 | 32.6% | +121% | +18.2% | 0.414 | 0.633 | FAIL |
| BNB_1d | 45 | 44.4% | +1327% | +332.6% | 0.300 | 0.996 | PASS |
| BTC_4h * | 87 | 37.9% | +102% | -29.5% | 0.214 | 0.160 | FAIL |
| ETH_4h * | 118 | 50.8% | +226% | +1.7% | 0.000 | 0.355 | FAIL |
| SOL_4h * | 131 | 35.1% | +235% | -9.5% | 0.071 | 0.299 | FAIL |
| XRP_4h * | 93 | 45.2% | +403% | +123.8% | 0.414 | 0.924 | FAIL |
| BNB_4h * | 112 | 33.0% | +178% | +27.9% | 0.286 | 0.618 | FAIL |
* second run, same rules with different exit logic — retested from scratch rather than tuned into shape.
Read one row properly
BTC 4h * — the clearest illustration of why a good backtest proves nothing:
In-sample: +101.6%
Out-of-sample: −29.5%
Same rules. Same market. The only difference is that the second number comes from data the optimiser was never allowed to see. 87 out-of-sample trades — not a small sample that got unlucky.
Its PBO is 0.214, which passes. Its DSR is 0.160, which fails hard. That combination has a specific meaning: the configuration is not obviously curve-fit, but the return is not distinguishable from what you would expect to find by searching a lot of parameter combinations. It is the search that produced the number, not the edge.
What actually kills them
Of the 12 failures, 12 passed the PBO gate and were killed by DSR alone. That is the pattern worth taking away: overfitting in the obvious sense is rare. What is common is a Sharpe that looks fine until you account for how many things were tried before it appeared.
Three configurations passed both gates: BTC daily, SOL daily, BNB daily. All three are daily, none are 4-hour. That is the sort of finding you cannot get from a backtest — only from testing the backtest.
What would change a verdict
- More out-of-sample folds — several of these have 2 or 3. More history narrows the confidence, in either direction.
- Fewer configurations tried. DSR penalises the search itself, so a rule with two parameters is judged more kindly than one with six.
- Nothing else. Not a nicer chart, not a better narrative, not a longer in-sample run.
Your report looks like this. Same gauntlet, same thresholds, fixed before your data is touched. If your rules fail, the report says so and tells you which gate they failed at — that is the deliverable, not a consolation.
→ Send one rule through it · back to the offer
Numbers generated by my own walk-forward pipeline. Fees 0.2% per trade included. Verdict logic and thresholds are the same code that would run on your rules.