Benchmark results, in full

Where these numbers came from

Run

actions/runs/32396472073

Measured

2026-08-20

Commit

6cae68628325

Harness

benchmarks/vs_vectorbt

A GitHub-hosted runner is a small shared machine, which is the point: it is the floor rather than the ceiling, and anyone can rent the identical one for free. Absolute times on a workstation are lower; the ratios are what travel.

One backtest, start to finish

Median wall time of a single call, the engines interleaved over 3 repetitions so a slow moment on the runner hits all of them equally. Data loading is excluded and the warmup call is discarded, so this is the simulation itself. Every row below produced identical results across the engines: a row where they disagree is never timed here, it is listed further down instead.

Ratios are the median of the per-repetition ratios against manifoldbt, not a ratio of medians. A ~ marks a point whose interquartile range exceeded 15% of its median: indicative only.

The parameter sweep that validates a strategy

One backtest tells you almost nothing; the grid around it is the actual job. Manifold-BT runs a grid as one native fan-out call, with the data loaded once and the combinations spread over every core. The other two engines have no sweep API, so a grid there is a Python loop paying a full backtest per cell. Wall time for the whole grid, and the peak RAM each engine adds running it, measured one engine per process:

Every grid is checked cell by cell before any of it is timed: the engines are joined on the parameter pair they actually ran, not on position, and the worst disagreement in the grid is what the gate sees. 3 repetitions per point. RAM is probed in a separate single-engine process, because interleaving is what makes a memory reading worthless: a dash there means the probe did not run for that engine at that point, not that it used nothing.

Where a challenger was not run

• vectorbt, 20,000 combinations at 20,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file. An empty cell in a speed table reads as a defeat, so the reason is written out instead.
• vectorbt, 10,000 combinations at 200,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file. An empty cell in a speed table reads as a defeat, so the reason is written out instead.

Not run, and why

• raptorbt on ema_rsi_fees. Two blockers, either one sufficient. Sizing: raptorbt has no fixed-quantity mode. `position_sizes` is a fraction of equity (measured: 0.5 buys exactly half the equity of the bar before the entry), `lot_size` rounds a computed size down to a multiple, and `alloted_capital` fixes the notional, not the quantity. Reproducing `units=5` would mean feeding a fraction derived from an equity curve that does not exist until the run is over. Indicator: `raptorbt.ema` seeds on a simple mean of the first `period` bars and emits from bar `period-1`, while manifoldbt seeds on the first observation and emits from bar 0 (measured: 11 leading NaN for span 12, and ema[11] equal to sma(12)[11] to the last bit). The two are the same recursion with a different warmup, so the signal differs early and the round-trip count with it. Its `sma` and `rsi` do match, which is why the other three workloads run.
• raptorbt on sma_cross_costs. Same blocker as the other fixed-quantity workload: raptorbt has no units sizing, and a cost model on top of a fraction of equity compares policy rather than arithmetic. See `ema_rsi_fees` for the measurements behind that.
• raptorbt on multi_asset. No multi-instrument portfolio on the entry point this harness drives: `run_single_backtest` is one instrument, and `run_multi_backtest` broadcasts over strategies rather than over assets. 0.9.0 does add `run_portfolio_backtest`, but its allocation model is a different one and would need its own parity work before any timing from it could be published. The units blocker applies here too.

Cold start and memory

The two costs a timing loop hides. Cold start is what a fresh interpreter pays before the first result exists, which is what an agent, a CI job or a one-shot script actually feels. Memory is measured in a dedicated single-engine process, never in the interleaved timing run, where the peak sampled during one engine's call would include every other engine's allocations.

What the engines agreed on

A speed number is worth nothing if the engines computed different things, so every workload is compared before it is timed: final equity against starting capital, round-trip count, fees. 13 of the 16 measured points came back exact. 3 carry a known, documented divergence and are kept out of the headline. None disagreed unexpectedly.

• vectorbt. Re-entry on the exit bar. When a bracket fires intrabar and the entry condition still holds at that bar's close, manifoldbt books two orders on that bar (the stop or target exit, then a fresh entry at the close); vectorbt processes one order per bar and re-enters on the next bar instead. Neither is wrong, and on controlled bars the bracket fills themselves match exactly (see the cross-engine parity suite shipped with the library). The harness counts the affected round-trips so the size of the divergence is measured, not asserted.
• raptorbt. Same fork in the road, taken further: raptorbt does not re-arm at all. Once a bracket closes a position, the entry level being still true is not enough to open another one; it waits for the level to go false and true again. So on the same bars the three engines book a different number of round-trips from the same signal, manifoldbt re-entering on the exit bar, vectorbt on the next bar, raptorbt not until the next crossing. The count of affected round-trips is measured rather than asserted, and it is the same population for both challengers.
• On bracket_sl_tp at 100,000 bars, 925 of 2,309 manifoldbt round-trips re-enter on the exit bar (40%), from 632 stop and 388 target exits. raptorbt books 1,384.

Run it yourself

A self-reported benchmark is worth what its reproduction costs. Fork the repository and press "Run workflow" on Benchmark vs vectorbt and raptorbt to get this table on your own runner, or run the harness locally:

# Every engine comes from PyPI, exactly as you would install it.
git clone https://github.com/manifoldbt/manifoldbt
cd manifoldbt/benchmarks/vs_vectorbt
pip install manifoldbt
pip install -r requirements-lock.txt

# Backtests, cold start and memory
python bench.py --bars 100000 1000000 10000000 --reps 3 \
  --cold-start-reps 3 --memory-bars 2000000

# The parameter grids, which need a licence (see below)
python bench.py --workloads sma_cross --bars 100000 \
  --sweep 20000:5000 20000:20000:oos 200000:10000:oos --sweep-reps 3

The sweep points need a licence: an unlicensed fan-out call waits out a fixed interval before any work starts, so the stopwatch would be timing the wait rather than the engine. Everything else runs on the free tier.