Why AI-written backtests lie

1. It traded on a bar it could not have seen

Look-ahead is the oldest lie in backtesting and the one an agent writes most naturally, because in a DataFrame every row is already there. The crude form is filling at a price that precedes the close that produced the signal: with the default execution at the close, deciding and filling on the same close is a valid convention, but with any other fill price a delay of zero fills before the information existed. The subtle forms are the ones that survive review. A daily trend filter evaluated on the day that is still forming, so the hourly strategy knows where today closes. A z-score normalised with the mean and standard deviation of the whole series, so every bar knows the future distribution. A rolling statistic centred rather than trailing, one keyword away in pandas.

In the bench, the task built to catch this (a daily signal on hourly execution) went to 0 of 3 for Haiku writing with the engine installed and no reference, at low, medium and high reasoning effort alike, and to 2, 3 and 3 of 3 once it could read that a completed daily bar becomes available at the start of the next day and not before. Sonnet did not need the help on that task; the smaller the model, the more it guesses. The check does not read the code. It cuts the future off and asks whether the past changes:

from manifoldbt.diagnostics import detect_lookahead

# Reruns the strategy on truncated and extended windows and compares the
# trades to the full run. A trade that changes when the future is cut off
# was made with the future in hand.
report = detect_lookahead(strategy, config, store)
print(report)            # PASS or FAIL, with the sub-tests that failed
report.assert_clean()    # raise instead, for a script that must not pass silently

detect_lookahead is a Pro diagnostic, alongside check_exposure_stability, which asks the same question of position sizes across windows. On any tier, the trade log carries signal_timestamp and execution_timestamp per fill, and a strategy whose two columns are equal on signal fills, with an execution price other than the close, has already told you what it did.

2. It decided the fill it would have got

Left alone, an agent writes the fill it needs: at the close it saw, for free, in full. Real fills have conventions, and the conventions are where the money is. A stop that is gapped through does not fill at its level, it fills at the open. A take-profit that the bar merely touched may or may not have been yours; a resting limit order that fills the moment price reaches it assumes a queue position nobody proved. A reversal is one fill whose quantity is both legs added together, and it pays fees on both. Skip any of these and the strategy earns a spread that does not exist.

This was the clearest line in the whole bench. A Donchian breakout with a stop and a target went 0 of 3 on Haiku and 1 of 3 on Sonnet without the reference, and 3 of 3 with it on both. The runs that missed returned −6%, −6% and −18% against a reference of −15%: same rule, same data, three different ideas of what a stop does. The two settings that make it honest, and the two counters that say how much it mattered:

# Two settings, then two counters.
config = mbt.BacktestConfig(
    ...,
    execution=mbt.ExecutionConfig(
        signal_delay=1,                       # bar t decides, bar t+1 fills
        fill_model={"passive_fill": "traverse"},  # a resting order fills when
    ),                                        # the bar trades THROUGH it
    fees=mbt.FeeConfig(taker_fee_bps=5.0, maker_fee_bps=5.0),
)
result = mbt.run(strategy, config, store)

f = result.fill_fragility
print(f["maker_fills"], f["touch_only_fills"])
# touch_only_fills counts the fills that exist ONLY because the default
# 'touch' model booked an order when the bar merely reached its level.
# A large share of maker_fills here means the edge was a queue assumption.

fill_fragility is a believability number, not a performance number. Run the strategy with the default touch model, read how many maker fills exist only because of it, then rerun with traverse. What survives is the strategy; what evaporated was the assumption. For a strategy with brackets, accuracy=True runs the fills on one-minute bars, so a stop and a target hit inside the same hour are resolved in the order they happened.

3. It reported the best cell of the grid

Ask an agent to optimise and it will sweep the grid and hand you the winner, as a strategy. With a few hundred combinations, the best Sharpe in the grid is high by construction; that is what a maximum over noise looks like. Bailey and López de Prado named the correction the deflated Sharpe ratio, and the intuition is enough here: the more cells you searched, the less the top one means. The question is not which cell won, it is how much of the grid agrees with it.

The bench had a walk-forward task, and its lesson was about selection rather than search. Selecting each fold’s parameter on the in-sample Sharpe including the warm-up bars, instead of the tradable part of the window, changed the choice on 2 folds out of 10 and moved the stitched out-of-sample return from −10.35% to +22.10%. Same grid, same data, one detail of where the score is measured. Three tools answer the selection question directly:

# The grid, then how much of it agrees with the winner.
sweep = mbt.run_sweep_2d(strategy, {
    "x_param": "fast", "x_values": list(range(5, 60, 5)),
    "y_param": "slow", "y_values": list(range(20, 200, 20)),
    "metric": "sharpe",
}, config, store)
mbt.plot.heatmap_2d(sweep, zones=True, show=True)   # plateau, not peak

# One parameter, one score: how far the metric moves around the chosen value
stab = mbt.run_stability(strategy, {
    "param_name": "fast", "values": list(range(10, 31)), "metric": "sharpe",
}, config, store)
print(stab["stability_score"], stab["mean_metric"], stab["std_metric"])

The heatmap highlights the plateau-optimal cell after Gaussian smoothing rather than the raw peak, and zones=True draws the regions where the metric stays stable across neighbours. The stability score is one number for one parameter. And walk-forward is the honest version of the whole exercise: optimise in-sample, score out of sample, fold by fold, with the in-sample score taken where the strategy could actually trade. The free Community tier caps a session at 256 combinations across its sweeps; stability and walk-forward are Pro.

The number the other numbers do not show

One failure in the bench was not fixed by any documentation. On the multi-asset task the agent returned 42,570 fills where the reference had 1,413, with return, Sharpe and drawdown all within 2% of the reference. It had rebalanced every bar instead of holding a quantity fixed between two changes of target. Every performance metric said the strategy was right; only the fill count said it was a different strategy, and a different strategy has different costs the moment it leaves the backtest. Read the trade count first. total_trades counts fills, an entry and its exit are two; round_trips counts completed trades. If either is far from what the rule implies, the rule is not what ran.

Making the agent run the checks

None of this requires distrusting the agent, only asking it for more than four numbers. Ask for the trade count with the metrics. Ask for a one-bar signal delay and fees on both sides, and say so in the prompt, because left to choose it will pick the frictionless version. Ask for detect_lookahead and fill_fragility before any optimisation, and for a stability score or a walk-forward after it, instead of a winner. With the engine’s reference connected over MCP the agent reads these rules before it writes, which is the difference the bench measured; the Claude Code guide walks through the setup and the prompt. The strategy an agent writes in a minute is not the problem. The strategy nobody checked is.

Frequently asked questions