Benchmark results, in full
Apart from the GPU timings, every number on this page comes from one of two public runs of benchmark workflows in the engine repository: the first part times bar backtests and parameter grids, the second market making on order-book data. Both install each engine from PyPI on a GitHub-hosted runner, the same wheel a user installs, then compare results before they compare speed. At 10,000,000 bars one backtest with the full metric suite costs 210.7 ms, 391x faster than vectorbt and 5.8x faster than raptorbt. On a grid of 10,000 combinations at 200,000 bars, the gap over raptorbt widens to 36.2x, because only one of these engines has a sweep API.
This page is the whole result, not the flattering half: the workloads where the engines diverge, the grids a challenger could not hold in memory, and the points too noisy to quote are all here. Nothing below has to be taken on trust: the run that produced it is public, log and job summary included, and so is the harness that measured it: workload definitions, parity checks, and the method it all rests on. Market making and HFT has its own run and its own section, further down. The GPU timings come from a workflow of their own, which runs the same parameter grids on a card rented from Modal, since GitHub's runners have none: they are right after the sweep table, where 150,000 combinations at 200,000 bars run 25.6x faster on the card than on the runner.
Where the bar-engine numbers came from
- Measured
- 2026-11-08
- Commit
- 05d1ca42e9f1
- Harness
- benchmarks/vs_vectorbt
- Machine
- AMD EPYC 9V74 80-Core Processor, 4 logical cores, 16.8 GB RAM
- OS
- Linux 7.0.0-1019-azure x86_64
- Python
- 3.12.16
- Engines
- manifoldbt 0.33.0, vectorbt 0.28.4, raptorbt 0.9.0, nautilus_trader 1.231.0, numpy 2.4.3, numba 0.64.0, pandas 2.3.3
- Machine
- AMD EPYC 9V74 80-Core Processor, 4 logical cores, 16.8 GB RAM
- OS
- Linux 7.0.0-1019-azure x86_64
- Python
- 3.12.16
- Engines
- manifoldbt 0.33.0, vectorbt 0.28.4, raptorbt 0.9.0, nautilus_trader 1.231.0, numpy 2.4.3, numba 0.64.0, pandas 2.3.3
A GitHub-hosted runner is a small shared machine, which is the point: it is the floor rather than the ceiling, and anyone can rent the identical one for free. Absolute times on a workstation are lower; the ratios are what travel.
One backtest, start to finish
Median wall time of a single call, the engines interleaved over 7 repetitions so a slow moment on the runner hits all of them equally. Data loading is excluded and the warmup call is discarded, so this is the simulation itself. Every row below produced identical results across the engines: a row where they disagree is never timed here, it is listed further down instead.
| Workload | Bars | manifoldbt | vectorbt | raptorbt | NautilusTrader |
|---|---|---|---|---|---|
| sma_cross | 100,000 | 4.3 ms | 139.8 ms32.5x slower ~ | 7.7 ms1.8x slower ~ | 4.20 s993x slower ~ |
| sma_cross_metrics | 100,000 | 4.3 ms | 700.8 ms159x slower ~ | 7.6 ms1.8x slower ~ | 4.22 s974x slower ~ |
| sma_cross | 1,000,000 | 22.8 ms | 1.65 s73.5x slower | 80.0 ms3.5x slower | not run> 100,000 bars |
| sma_cross_metrics | 1,000,000 | 24.2 ms | 6.70 s277x slower | 93.1 ms3.9x slower | not run> 100,000 bars |
| sma_cross | 10,000,000 | 197.8 ms | 20.68 s104x slower | 1.22 s6.1x slower | not run> 100,000 bars |
| sma_cross_metrics | 10,000,000 | 210.7 ms | 82.22 s391x slower | 1.22 s5.8x slower | not run> 100,000 bars |
| ema_rsi_fees | 100,000 | 5.9 ms | 147.8 ms24.9x slower | not run | 9.07 s1,543x slower |
| ema_rsi_fees | 1,000,000 | 36.5 ms | 1.93 s51.1x slower | not run | not run> 100,000 bars |
| sma_cross_costs | 100,000 | 4.0 ms | 142.2 ms34.8x slower ~ | not run | 4.21 s1,048x slower ~ |
| sma_cross_costs | 1,000,000 | 24.7 ms | 1.96 s74.1x slower ~ | not run | not run> 100,000 bars |
| sma_cross_costs | 10,000,000 | 202.6 ms | 21.39 s106x slower | not run | not run> 100,000 bars |
| multi_asset | 100,000 | 15.4 ms | 171.8 ms11.1x slower | not run | 38.22 s2,464x slower |
| multi_asset | 1,000,000 | 130.9 ms | 2.05 s15.7x slower ~ | not run | not run> 100,000 bars |
Under each challenger's time: how much slower or faster it is than manifoldbt, as the median of the per-repetition ratios, not a ratio of medians. A ~ marks a point whose interquartile range exceeded 15% of its median: indicative only.
The parameter sweep that validates a strategy
One backtest tells you almost nothing; the grid around it is the actual job. Manifold-BT runs a grid as one native fan-out call, with the data loaded once and the combinations spread over every core. The other three engines have no sweep API, so a grid there is a Python loop paying a full backtest per cell. Wall time for the whole grid, with each challenger's time against manifoldbt under it:
| Bars | Combinations | manifoldbt | vectorbt | raptorbt | NautilusTrader |
|---|---|---|---|---|---|
| 20,000 | 5,000 | 236.1 ms | 4.12 s17.4x slower | 5.15 s21.8x slower | not run |
| 20,000 | 20,000 | 900.0 ms | not run | 20.27 s22.5x slower | not run |
| 200,000 | 10,000 | 3.53 s | not run | 127.81 s36.2x slower | not run |
| 200,000 | 150,000 | 72.99 s | not run | not run | not run |
Peak memory each engine adds running the grid
| Bars | Combinations | manifoldbt | vectorbt | raptorbt | NautilusTrader |
|---|---|---|---|---|---|
| 20,000 | 5,000 | 38 MB | 2.5 GB | 37 MB | - |
| 20,000 | 20,000 | 80 MB | - | 63 MB | - |
| 200,000 | 10,000 | 345 MB | - | 370 MB | - |
| 200,000 | 150,000 | 834 MB | - | - | - |
Every grid is checked cell by cell before any of it is timed: the engines are joined on the parameter pair they actually ran, not on position, and the worst disagreement in the grid is what the gate sees. 2 repetitions per point. RAM is probed in a separate single-engine process, because interleaving is what makes a memory reading worthless: a dash there means the probe did not run for that engine at that point, not that it used nothing.
Where a challenger was not run
An empty cell in a speed table reads as a defeat, so the reason is written out instead.
- NautilusTrader, 5,000 combinations at 20,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 0.96 s per backtest at 20,000 bars (mean over 16 cells spread across this grid), which puts this grid of 5,000 combinations at about 80 minutes per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.
- vectorbt, 20,000 combinations at 20,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file.
- NautilusTrader, 20,000 combinations at 20,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 1.12 s per backtest at 20,000 bars (mean over 16 cells spread across this grid), which puts this grid of 20,000 combinations at about 6.2 hours per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.
- vectorbt, 10,000 combinations at 200,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file.
- NautilusTrader, 10,000 combinations at 200,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 10.23 s per backtest at 200,000 bars (mean over 16 cells spread across this grid), which puts this grid of 10,000 combinations at about 28.4 hours per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.
- vectorbt, 150,000 combinations at 200,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file.
- raptorbt, 150,000 combinations at 200,000 bars. raptorbt has no fan-out API for a parameter grid on one instrument, so a grid is a Python loop of full backtests. Measured: 11.9 ms per backtest at 200,000 bars (mean over 16 cells spread across this grid), which puts this grid of 150,000 combinations at about 30 minutes per call. That is past the time budget of the job. It is not run.
- NautilusTrader, 150,000 combinations at 200,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 18.68 s per backtest at 200,000 bars (mean over 16 cells spread across this grid), which puts this grid of 150,000 combinations at about 778.2 hours per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.
The same sweeps on a GPU
GitHub's runners have no GPU, so the grids above are timed on the CPU path only. A second workflow runs the same grids, on the same generated bars and with the same strategy, on an NVIDIA L4 rented from Modal, with the wheel installed from PyPI as everywhere else on this page. It times the card alone: the CPU column below is the runner's time from the table above, for the same grid and the same data. GPU acceleration is a Researcher feature.
- Measured
- 2026-11-08
- Card
- NVIDIA L4, 23034 MiB, driver 610.57.04
- Engine
- manifoldbt 0.33.0
| Bars | Combinations | GPU | CPU (runner) | Card vs runner | Bit-identical |
|---|---|---|---|---|---|
| 20,000 | 5,000 | 31.8 msfirst call 455.1 ms | 236.1 ms | 7.4x faster | 5,000 / 5,000 |
| 20,000 | 20,000 | 109.4 msfirst call 119.8 ms | 900.0 ms | 8.2x faster | 20,000 / 20,000 |
| 200,000 | 10,000 | 229.6 msfirst call 240.3 ms | 3.53 s | 15.4x faster | 10,000 / 10,000 |
| 200,000 | 150,000 | 2.85 sfirst call 2.99 s | 72.99 s | 25.6x faster | 150,000 / 150,000 |
GPU: medians of 7 repetitions, after the card is calibrated and each grid has run once untimed, since that first call compiles the kernels; it is shown under each time because it is what a first sweep costs. Every cell of the GPU grid is compared with the same grid run once on the CPU of the GPU machine, keyed by parameter pair, and the last column counts the cells equal to the bit. The ratio crosses two machines, a rented NVIDIA L4 and a four-vCPU GitHub runner: it says what the sweep costs on the runner the CPU numbers come from against what it costs on the card, not how the card compares with the CPU beside it. A ratio is only given when both runs measured the same version on the same data. A ~ marks a time whose interquartile range exceeded 15% of its median: indicative only.
The run is reproducible like the others: fork the repository, add a Modal token and a licence as secrets, and run Benchmark GPU sweeps (Modal), or run the harness on any machine with an NVIDIA card. Method and code: benchmarks/gpu.
Not run, and why
- NautilusTrader on
sma_cross,sma_cross_metrics,ema_rsi_fees,sma_cross_costs,multi_asset,bracket_sl_tp. Not run above 100,000 bars. NautilusTrader is event-driven: every bar goes through a simulated exchange and a Python strategy callback, so one call costs seconds per 100,000 bars and grows linearly with the series. Repeated and interleaved, a longer series does not fit the time budget of the job. The measured time at the ceiling is in the table, andbench.py --engine-max-bars nautilus_trader=Nlifts the ceiling for a local run. - raptorbt on
ema_rsi_fees. Two blockers, either one sufficient. Sizing: raptorbt has no fixed-quantity mode.position_sizesis a fraction of equity (measured: 0.5 buys exactly half the equity of the bar before the entry),lot_sizerounds a computed size down to a multiple, andalloted_capitalfixes the notional, not the quantity. Reproducingunits=5would mean feeding a fraction derived from an equity curve that does not exist until the run is over. Indicator:raptorbt.emaseeds on a simple mean of the firstperiodbars and emits from barperiod-1, while manifoldbt seeds on the first observation and emits from bar 0 (measured: 11 leading NaN for span 12, and ema[11] equal to sma(12)[11] to the last bit). The two are the same recursion with a different warmup, so the signal differs early and the round-trip count with it. Itssmaandrsido match, which is why the other three workloads run. - raptorbt on
sma_cross_costs. Same blocker as the other fixed-quantity workload: raptorbt has no units sizing, and a cost model on top of a fraction of equity compares policy rather than arithmetic. Seeema_rsi_feesfor the measurements behind that. - raptorbt on
multi_asset. No multi-instrument portfolio on the entry point this harness drives:run_single_backtestis one instrument, andrun_multi_backtestbroadcasts over strategies rather than over assets. 0.9.0 does addrun_portfolio_backtest, but its allocation model is a different one and would need its own parity work before any timing from it could be published. The units blocker applies here too.
Cold start and memory
The two costs a timing loop hides. Cold start is what a fresh interpreter pays before the first result exists, which is what an agent, a CI job or a one-shot script actually feels. Memory is measured in a dedicated single-engine process, never in the interleaved timing run, where the peak sampled during one engine's call would include every other engine's allocations.
| manifoldbt | vectorbt | raptorbt | NautilusTrader | |
|---|---|---|---|---|
| Fresh process to first backtest | 0.41 s | 4.78 s | 0.31 s | 1.54 s |
| Minus the 0.29 s bare interpreter | 0.12 s | 4.49 s | 0.03 s | 1.25 s |
| RAM added by the run, per million bars | under 0.1 MB | 139 MB | 5.2 MB | 6.7 MBat 100,000 bars |
Cold start is timed on sma_cross at 20,000 bars. Memory is measured on 2,000,000 bars. An engine held to a ceiling is measured at it and scaled to a million bars like the others; its cell says at how many.
What the engines agreed on
A speed number is worth nothing if the engines computed different things, so every workload is compared before it is timed: final equity against starting capital, round-trip count, fees. 13 of the 16 measured points came back exact. 3 carry a known, documented divergence and are kept out of the headline. None disagreed unexpectedly.
| Workload | Bars | manifoldbt | vectorbt | raptorbt | NautilusTrader |
|---|---|---|---|---|---|
| bracket_sl_tp | 100,000 | 7.2 ms | 142.9 ms | 6.3 ms | 9.54 s |
| bracket_sl_tp | 1,000,000 | 51.7 ms | 2.45 s | 61.9 ms | not run |
| bracket_sl_tp | 10,000,000 | 445.8 ms | 24.22 s | 1.26 s | not run |
- vectorbt. Re-entry on the exit bar. When a bracket fires intrabar and the entry condition still holds at that bar's close, manifoldbt books two orders on that bar (the stop or target exit, then a fresh entry at the close); vectorbt processes one order per bar and re-enters on the next bar instead. Neither is wrong, and on controlled bars the bracket fills themselves match exactly (see the cross-engine parity suite shipped with the library). The harness counts the affected round-trips so the size of the divergence is measured, not asserted.
- raptorbt. Same fork in the road, taken further: raptorbt does not re-arm at all. Once a bracket closes a position, the entry level being still true is not enough to open another one; it waits for the level to go false and true again. So on the same bars the three engines book a different number of round-trips from the same signal, manifoldbt re-entering on the exit bar, vectorbt on the next bar, raptorbt not until the next crossing. The count of affected round-trips is measured rather than asserted, and it is the same population for both challengers.
- NautilusTrader. Re-entry sizing. Entries, exits and fill prices are the same trades as the reference's, round-trip for round-trip: the stop and the target are native contingent orders, they fill at their own levels, and because the simulated exchange runs before the strategy sees the bar, a position stopped out intrabar is re-entered at that bar's close just as the reference does. What differs is the quantity of that re-entry. The reference sizes it from the equity at the close, counting the position it has just closed at the close price, so it buys back the quantity it held; Nautilus sizes it from the cash balance after the stop or target has been booked at its own price. Neither is wrong, the two quantities differ by the gap between the exit price and the close, and that gap is carried into every later trade. The harness measures it (re-entries on the exit bar, and how far each one's quantity sits from the position closed on that bar) instead of asserting it.
- On bracket_sl_tp at 100,000 bars, 922 of 2,303 manifoldbt round-trips re-enter on the exit bar (40%), from 630 stop and 387 target exits. raptorbt books 1,381. NautilusTrader books 2,303. Of those, 922 re-enter on the exit bar, with a quantity that sits a median of 2.7e-4 (at most 1.1e-3), in relative terms, away from the position closed on that bar.
Run it yourself
A self-reported benchmark is worth what its reproduction costs. Fork the repository and press "Run workflow" on Benchmark vs vectorbt, raptorbt and NautilusTrader to get this table on your own runner, or run the harness locally:
# Every engine comes from PyPI, exactly as you would install it.
git clone https://github.com/manifoldbt/manifoldbt
cd manifoldbt/benchmarks/vs_vectorbt
pip install manifoldbt
pip install -r requirements-lock.txt
# Backtests, cold start and memory
python bench.py --bars 100000 1000000 10000000 --reps 7 \
--cold-start-reps 3 --memory-bars 2000000
# The parameter grids, which need a licence (see below)
python bench.py --workloads sma_cross --bars 100000 \
--sweep 20000:5000 20000:20000:oos=vectorbt 200000:10000:oos=vectorbt 200000:150000:oos=vectorbt,raptorbt --sweep-reps 2The sweep points need a licence: an unlicensed fan-out call waits out a fixed interval before any work starts, so the stopwatch would be timing the wait rather than the engine. Everything else runs on the free tier.
Market making and HFT, engine by engine
A second public benchmark, on order-book data: manifoldbt, hftbacktest and NautilusTrader run the same trading rules, written once, on one pinned day of public market data. The harness checks that the engines did the same work first, and only then reports how long each took. With market making, counting fills is not enough to know whether they did: an engine that serves an order in several smaller pieces shows more fills for the same executed quantity, and an engine that never serves a print that went through a quote shows fewer for the same market.
Parity is therefore read in three layers: the actions (which quotes were sent, at which wake-up, on which side, at which price), the verdicts (for a quote both engines sent, was it served, and for how much) and the totals (fills, executed quantity). manifoldbt is the reference: each challenger is compared with it, never with the other challenger, and a time beside a name is that engine's time against manifoldbt's, as a ratio of medians. A ~ marks a median whose interquartile range exceeded 15%: indicative only. Where the engines differ, the mechanisms are listed under why the engines differ.
Where the market-making numbers came from
- Measured
- 2026-11-08
- Commit
- 05d1ca42e9f1
- Market data day
- 2025-08-13, 3 interleaved repetitions
- Harness
- benchmarks/vs_hft
- Order book archive (Bybit spot BTCUSDT, 200 levels)
- https://quote-saver.bycsi.com/orderbook/spot/BTCUSDT/2025-08-13_BTCUSDT_ob200.data.zip, 89.9 MB
- 63e1f91c9b3e28fd0189a0bf1773eadf0e02cc0d63c1608d4403013da848bc38
- Trade archive (Bybit spot BTCUSDT)
- https://public.bybit.com/spot/BTCUSDT/BTCUSDT_2025-08-13.csv.gz, 5.1 MB
- 5a9277a3c73f93af1811917a29cf723fc34b5d4f1a17868dbd90d5fa26dd671c
- Prepared arrays both engines read
- 70daffab0e402e39de58db80aa28bcbcef8592a1eaea5eba4033bcc0386c3759
- Nasdaq TotalView-ITCH extract, AAPL
- AAPL.itch, 40.3 MB
- 6d66ec7f49fc61971f4617208d179258ebf26a976a29a10255dfb7574fd07106
The run downloads these from the source and stops if a fingerprint changed. Nothing is redistributed. Two runs that print the same fingerprints read the same bytes.
- Machine
- 4 logical CPUs (2 physical), 16.8 GB RAM
- OS
- Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 x86_64
- Python
- 3.12.16
- Engines
- manifoldbt 0.33.0, hftbacktest 2.4.4, nautilus_trader 1.231.0, numba 0.68.0, numpy 2.2.6, pandas 3.1.0, pyarrow 26.0.0
- manifoldbt library SHA-256
- 84d4cb78e690b29b3d6e41e15985b8071eee7d8680a8aa9e293faa63edae1de1
- Machine
- 4 logical CPUs (2 physical), 16.8 GB RAM
- OS
- Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 x86_64
- Python
- 3.12.16
- Engines
- manifoldbt 0.33.0, hftbacktest 2.4.4, nautilus_trader 1.231.0, numba 0.68.0, numpy 2.2.6, pandas 3.1.0, pyarrow 26.0.0
- manifoldbt library SHA-256
- 84d4cb78e690b29b3d6e41e15985b8071eee7d8680a8aa9e293faa63edae1de1
Order books by price
Bybit spot BTCUSDT, one pinned day. The best bid/ask and the top 25 levels are cut from the same replayed states.
mm_ladder_l2
Two post-only quotes every 100 ms, one level further back per clip of inventory held, inventory capped at 5 clips. Reads 25 levels a side and the trades.
25 levels a side + trades · 5 / 5 / 5 / 2 ms
| Engine | Strategy run | Replay only | Added RAM | First run in a process |
|---|---|---|---|---|
| manifoldbt | 541.0 ms | 178.0 ms | 570 MB | 936.0 ms |
| hftbacktest | 1.53 s2.8x slower | 384.0 ms | 1.6 GB | 6.31 s |
| NautilusTrader | 133.25 s246x slower | 42.42 s | 1.4 GB | 133.25 s |
- NautilusTrader: 124.08 s in the engine, 7.89 s building its data objects; both are inside the strategy run.
- Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 20.0 ms, hftbacktest 1.91 s.
- Actions part at wake-up 310, verdicts differhftbacktest against manifoldbt
- hftbacktest sent 133,194 quotes and manifoldbt 133,494; 124,951 of the 130,505 they share are at the same price.
- Verdicts on the shared quotes: served by both 27,222, only by hftbacktest 6, only by manifoldbt 49; the same executed quantity on 27,073 of the 27,222 served by both.
- Quotes served: 28,897 (hftbacktest), 28,766 (manifoldbt). Fills booked: 28,897 against 36,579; a fill is a piece of an order, so this is not a count of quotes served.
- Actions part at wake-up 44, verdicts differNautilusTrader against manifoldbt
- NautilusTrader sent 116,412 quotes and manifoldbt 133,494; 72,848 of the 102,871 they share are at the same price.
- Verdicts on the shared quotes: served by both 18,361, only by NautilusTrader 95, only by manifoldbt 293; the same executed quantity on 18,104 of the 18,361 served by both.
- Quotes served: 25,663 (NautilusTrader), 28,766 (manifoldbt). Fills booked: 25,663 against 36,579; a fill is a piece of an order, so this is not a count of quotes served.
mm_touch_bbo
The same maker on a best bid/ask feed: it can only join the touch.
best bid/ask + trades · 5 / 5 / 5 / 2 ms
| Engine | Strategy run | Replay only | Added RAM | First run in a process |
|---|---|---|---|---|
| manifoldbt | 376.0 ms | 144.0 ms | 484 MB | 597.0 ms |
| hftbacktest | 405.0 ms1.1x slower | 64.0 ms | 369 MB | 5.24 s |
| NautilusTrader | 62.07 s165x slower | 7.36 s | 1011 MB | 62.07 s |
- NautilusTrader: 59.54 s in the engine, 1.33 s building its data objects; both are inside the strategy run.
- Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 19.0 ms, hftbacktest 844.0 ms.
- Actions part at wake-up 488, verdicts differhftbacktest against manifoldbt
- hftbacktest sent 76,682 quotes and manifoldbt 76,847; 75,198 of the 75,198 they share are at the same price.
- Verdicts on the shared quotes: served by both 30,102, only by hftbacktest 17, only by manifoldbt 24; the same executed quantity on 29,966 of the 30,102 served by both.
- Quotes served: 31,023 (hftbacktest), 30,731 (manifoldbt). Fills booked: 31,023 against 43,664; a fill is a piece of an order, so this is not a count of quotes served.
- Actions part at wake-up 37, verdicts differNautilusTrader against manifoldbt
- NautilusTrader sent 68,913 quotes and manifoldbt 76,847; 63,539 of the 63,539 they share are at the same price.
- Verdicts on the shared quotes: served by both 25,123, only by NautilusTrader 998, only by manifoldbt 631; the same executed quantity on 25,045 of the 25,123 served by both.
- Quotes served: 29,502 (NautilusTrader), 30,731 (manifoldbt). Fills booked: 29,502 against 43,664; a fill is a piece of an order, so this is not a count of quotes served.
flow_taker_bbo
Follow the aggressor imbalance of the last 34 s (arm at +/-0.9), market orders, flip on the opposite crossing. Trade clock.
best bid/ask + trades · 20 ms order delay
| Engine | Strategy run | Added RAM | First run in a process |
|---|---|---|---|
| manifoldbt | 87.0 ms | 366 MB | 305.0 ms |
| hftbacktest | 83.0 ms1.0x faster | 329 MB | 3.10 s |
| NautilusTrader | 9.83 s113x slower | 646 MB | 9.83 s |
- NautilusTrader: 7.51 s in the engine, 1.32 s building its data objects; both are inside the strategy run.
- Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 20.0 ms, hftbacktest 849.0 ms.
- Same actionshftbacktest against manifoldbt
- 364 decisions at the same times and on the same sides; 364 orders identical.
- Fills 364 against 364; executed quantity 7.27 against 7.27; notional -0.034 bp from manifoldbt.
- Same actionsNautilusTrader against manifoldbt
- 364 decisions at the same times and on the same sides; 364 orders identical.
- Fills 447 against 364; executed quantity 7.27 against 7.27; notional +0.006 bp from manifoldbt.
The open-loop probe
Not a strategy but an instrument. Its quotes depend on the market only, so every quote gets a verdict (served or not) that does not depend on what the engine did before. That isolates the queue and fill model of each engine.
probe_l2_zero
Open-loop probe: one post-only quote a side at the touch, cancelled and posted again at every wake-up, no inventory, zero latency.
25 levels a side + trades · zero latency · window 10:00 to 11:00 UTC
| Engine | Strategy run | Added RAM | First run in a process |
|---|---|---|---|
| manifoldbt | 257.0 ms | 374 MB | 523.0 ms |
| hftbacktest | 86.0 ms3.0x faster | 373 MB | 4.91 s |
| NautilusTrader | 19.03 s74.1x slower | 933 MB | 19.03 s |
- NautilusTrader: 17.65 s in the engine, 348.0 ms building its data objects; both are inside the strategy run.
- Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 20.0 ms, hftbacktest 674.0 ms.
- Same actions, verdicts differhftbacktest against manifoldbt
- The same 71,994 quotes, at the same prices.
- Verdicts on the shared quotes: served by both 1,590, only by hftbacktest 0, only by manifoldbt 16; the same executed quantity on 1,561 of the 1,590 served by both.
- Quotes served: 1,590 (hftbacktest), 1,606 (manifoldbt). Fills booked: 1,590 against 1,792; a fill is a piece of an order, so this is not a count of quotes served.
- Actions differ on 4 of 71,998 quotes, verdicts differNautilusTrader against manifoldbt
- NautilusTrader sent 71,998 quotes and manifoldbt 71,994; 71,994 of the 71,994 they share are at the same price.
- Verdicts on the shared quotes: served by both 1,311, only by NautilusTrader 0, only by manifoldbt 295; the same executed quantity on 1,249 of the 1,311 served by both.
- Quotes served: 1,311 (NautilusTrader), 1,606 (manifoldbt). Fills booked: 1,311 against 1,792; a fill is a piece of an order, so this is not a count of quotes served.
probe_l2_nominal
The same open-loop probe under the 5 / 5 / 5 / 2 ms venue.
25 levels a side + trades · 5 / 5 / 5 / 2 ms · window 10:00 to 11:00 UTC
| Engine | Strategy run | Added RAM | First run in a process |
|---|---|---|---|
| manifoldbt | 256.0 ms | 374 MB | 524.0 ms |
| hftbacktest | 86.0 ms3.0x faster | 373 MB | 4.85 s |
| NautilusTrader | 20.02 s78.2x slower | 935 MB | 20.02 s |
- NautilusTrader: 18.65 s in the engine, 348.0 ms building its data objects; both are inside the strategy run.
- Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 18.0 ms, hftbacktest 684.0 ms.
- Actions differ on 2 of 71,996 quotes, verdicts differhftbacktest against manifoldbt
- hftbacktest sent 71,996 quotes and manifoldbt 71,994; 71,994 of the 71,994 they share are at the same price.
- Verdicts on the shared quotes: served by both 1,430, only by hftbacktest 1, only by manifoldbt 8; the same executed quantity on 1,408 of the 1,430 served by both.
- Quotes served: 1,431 (hftbacktest), 1,438 (manifoldbt). Fills booked: 1,431 against 1,567; a fill is a piece of an order, so this is not a count of quotes served.
- Actions differ on 4 of 71,998 quotes, verdicts differNautilusTrader against manifoldbt
- NautilusTrader sent 71,998 quotes and manifoldbt 71,994; 71,994 of the 71,994 they share are at the same price.
- Verdicts on the shared quotes: served by both 1,132, only by NautilusTrader 5, only by manifoldbt 306; the same executed quantity on 1,111 of the 1,132 served by both.
- Quotes served: 1,137 (NautilusTrader), 1,438 (manifoldbt). Fills booked: 1,137 against 1,567; a fill is a piece of an order, so this is not a count of quotes served.
Order by order
Nasdaq TotalView-ITCH: every order is visible, so the queue position is known rather than modelled.
mbo_join_mbo
Join the best bid and ask with a round lot, stand a side aside behind a crowded best or at the inventory bound. Nasdaq TotalView-ITCH, order by order (AAPL).
order by order · zero latency
| Engine | Strategy run | Replay only | Added RAM | First run in a process |
|---|---|---|---|---|
| manifoldbt | 488.0 ms | 326.0 ms | 699 MB | 730.0 ms |
| hftbacktest | 635.0 ms1.3x slower | 610.0 ms | 262 MB | 5.56 s |
| NautilusTrader | 26.60 s54.5x slower | 11.49 s | 737 MB | 26.60 s |
- NautilusTrader: 24.18 s in the engine, 1.78 s building its data objects; both are inside the strategy run.
- Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 26.0 ms, hftbacktest 140.0 ms.
- Actions part at wake-up 428, verdicts differhftbacktest against manifoldbt
- hftbacktest sent 20,124 quotes and manifoldbt 20,002; 19,215 of the 19,215 they share are at the same price.
- Verdicts on the shared quotes: served by both 4,790, only by hftbacktest 83, only by manifoldbt 174; the same executed quantity on 3,638 of the 4,790 served by both.
- Quotes served: 5,162 (hftbacktest), 5,170 (manifoldbt). Fills booked: 5,162 against 10,261; a fill is a piece of an order, so this is not a count of quotes served.
- Actions part at wake-up 53, verdicts differNautilusTrader against manifoldbt
- NautilusTrader sent 20,067 quotes and manifoldbt 20,002; 19,038 of the 19,038 they share are at the same price.
- Verdicts on the shared quotes: served by both 4,891, only by NautilusTrader 1,499, only by manifoldbt 0; the same executed quantity on 3,854 of the 4,891 served by both.
- Quotes served: 6,828 (NautilusTrader), 5,170 (manifoldbt). Fills booked: 6,828 against 10,261; a fill is a piece of an order, so this is not a count of quotes served.
The probe, quote by quote
Every quote of the probe is judged on its own: served or not. The counts are over the quotes both engines sent at the same price, so a difference in them is a difference in the queue and fill model and nothing else.
| Engine, latency | Served by both | Only the engine | Only manifoldbt | Neither |
|---|---|---|---|---|
| hftbacktestzero latency | 1,590 | 0 | 16 | 70,388 |
| NautilusTraderzero latency | 1,311 | 0 | 295 | 70,388 |
| hftbacktest5 / 5 / 5 / 2 ms | 1,430 | 1 | 8 | 70,555 |
| NautilusTrader5 / 5 / 5 / 2 ms | 1,132 | 5 | 306 | 70,551 |
Parameter sweeps
manifoldbt sweeps a quoting strategy through run_sweep (every combination keeps its order journal) and run_sweep_lite (metrics only): the book and the tape are loaded once and the combinations run in parallel inside the engine. hftbacktest and NautilusTrader have no sweep call, so what is timed is the loop of runs a user would write, and a pool of processes, each holding its own copy of the data, created and warmed beforehand and reported apart. The grids are nested: each contains the smaller ones.
Maker: order size x inventory bound x levels back on each side
window 10:00 to 12:00 UTC
- manifoldbt run_sweep_lite
- hftbacktest, 4 processes
- NautilusTrader, 4 processes
- dashed and hollow: grid time projected
| Combinations in the grid | 4 | 16 | 64 | 256 |
|---|---|---|---|---|
| manifoldbt run_sweep_lite | 33.9/s118.0 ms305 MB | 57.6/s278.0 ms315 MB | 68.8/s930.0 ms313 MB | 65.6/s3.90 s314 MB |
| manifoldbt run_sweep | 30.5/s131.0 ms323 MB | 50.0/s320.0 ms436 MB | 58.4/s1.09 s736 MB | 57.1/s4.49 s2.2 GB |
| hftbacktest, 4 processes | 15.8/s253.8 ms | 22.6/s707.0 ms | 23.7/s2.70 s | 24.5/s10.43 s |
| hftbacktest, one process | 9.6/s416.7 ms | 9.8/s1.63 s | 9.8/s6.56 s | 9.5/s26.91 s |
| NautilusTrader, 4 processes | 0.248/s16.13 s | 0.250/s64.1 s projected | 0.254/s4.2 min projected | 0.256/s16.7 min projected |
| NautilusTrader, one process | 0.114/s35.15 s | 0.118/s2.3 min projected | 0.120/s8.9 min projected | 0.120/s35.6 min projected |
- At the rate measured on the largest grid, one hour of this machine covers about 236,185 with manifoldbt run_sweep_lite, 88,321 with hftbacktest, 921 with NautilusTrader.
- Every pool of processes returned exactly what the loop of runs returned, combination by combination.
- NautilusTrader ran at most 8 combinations of a grid: its rate is measured on those and the time for the whole grid is projected from it, which is what the dashed segments and the "projected" cells say.
- manifoldbt: 28 of 28 sampled combinations came out of the sweep identical, to the bit, to the same combination run alone.
Order delay as an axis: delay x inventory bound x levels back
window 10:00 to 12:00 UTC
| Combinations in the grid | 16 |
|---|---|
| manifoldbt run_sweep_lite | 57.3/s279.0 ms315 MB |
| manifoldbt run_sweep | 49.8/s321.0 ms453 MB |
| hftbacktest, 4 processes | 23.3/s686.4 ms |
| hftbacktest, one process | 9.8/s1.63 s |
| NautilusTrader, 4 processes | 0.255/s62.8 s projected |
| NautilusTrader, one process | 0.116/s2.3 min projected |
- Every pool of processes returned exactly what the loop of runs returned, combination by combination.
- NautilusTrader ran at most 8 combinations of a grid: its rate is measured on those and the time for the whole grid is projected from it, which is what the dashed segments and the "projected" cells say.
- manifoldbt: 8 of 8 sampled combinations came out of the sweep identical, to the bit, to the same combination run alone.
Flow taker on the trade clock: flow threshold x order size
| Combinations in the grid | 12 |
|---|---|
| manifoldbt run_sweep | 24.4/s492.0 ms823 MB |
| hftbacktest, 4 processes | 22.6/s532.1 ms |
| hftbacktest, one process | 12.2/s985.2 ms |
| NautilusTrader, 4 processes | 0.184/s65.1 s projected |
| NautilusTrader, one process | 0.083/s2.4 min projected |
- Every pool of processes returned exactly what the loop of runs returned, combination by combination.
- NautilusTrader ran at most 6 combinations of a grid: its rate is measured on those and the time for the whole grid is projected from it, which is what the dashed segments and the "projected" cells say.
- manifoldbt: 4 of 4 sampled combinations came out of the sweep identical, to the bit, to the same combination run alone.
Why the engines differ
The three engines are set to the same venue wherever they can express it. Under the nominal venue an order, a cancel and a response each take 5 ms and the feed takes 2 ms; fees are 0 for a maker and 4 basis points for a taker; the wake-ups fall on the same instants; and each engine runs the closest equivalent of the others' queue model, listed in the harness README. hftbacktest runs with its no-partial-fill exchange, the benchmark's choice: an order is served whole at its first piece. NautilusTrader has no response delay, so the strategy emulates it. Two rules come from the engines' own semantics and are applied to all three: the wake-up clock starts at the first real book state inside the run, and the flow taker's trailing window of 34 s gives no signal until a whole window of tape has gone by.
Where they still differ, measured:
- Partial fills. Where an order is served in several pieces, the engines that serve in pieces show more fills for the same executed quantity, and a closed-loop strategy reacts to the first piece or to the whole order accordingly. This is the first place two closed-loop runs part, and each parity line gives the wake-up.
- NautilusTrader serves a resting order only through prints at its own price. With queue position on, a print through the level, a book that crosses the order and a shrinking of the displayed size ahead of it do not serve it. On the open-loop probe this shows as quotes that one engine serves and the other never does, and the probe table counts them.
- On an order-by-order market the right answer can be computed. The harness ships a replay of the day in explicit first-in first-out queues (
fifo_truth.py) that judges every quote the engines disagree on, engine by engine: how many it served although an exact queue had not served them yet, and how many it missed. It is a separate tool, and its verdicts are not part of the run published above. - Market orders. The taker's decisions and orders are compared exactly. The fill price is the price of the first print at or after the order's arrival in manifoldbt's trade-clock engine, and the book at the arrival in the other two; the measured gap of the executed notional is in the parity line.
The scenarios that reproduce each case on a book of a few lines, for all three engines, are in the harness README.
Run it yourself
The same commands run in GitHub Actions on public runners (bench-vs-hft.yml), so every number above has a run behind it. No account and no API key is needed for the data.
# Every engine comes from PyPI, exactly as you would install it.
git clone https://github.com/manifoldbt/manifoldbt
cd manifoldbt/benchmarks/vs_hft
pip install manifoldbt
pip install -r requirements-lock.txt
# One pinned day of public data, checked against its SHA-256
python prep_cli.py 2025-08-13
# Order books by price: the makers, the taker and the probe
python bench.py --day 2025-08-13 --reps 3 \
--workloads mm_ladder_l2 mm_touch_bbo flow_taker_bbo probe_l2_zero probe_l2_nominal \
--out results-books.json
# Order by order: the Nasdaq day file is read once and AAPL is kept
python itch_cli.py --day 2025-11-28 --symbols AAPL --out itch-extract
python bench.py --day 2025-08-13 --reps 3 \
--itch itch-extract/AAPL.itch --workloads mbo_join_mbo \
--out results-orders.json
# The maker sweeps
python sweep.py --family maker --day 2025-08-13 --window 36000 43200 \
--grids 4 16 64 256 --workers 4 --out results-sweeps-maker.json
python sweep_mbt.py --family maker --day 2025-08-13 --window 36000 43200 \
--grids 4 16 64 256 --workers 4 --identity 8 \
--out results-sweeps-mbt-maker.json
python report.py results-books.jsonKeep reading
Run your first backtest
Install Manifold-BT and reproduce the backtest above in seconds. The Rust core runs years of bars sub-second so you can sweep parameters instead of waiting.