Benchmark results, in full

Apart from the GPU timings, every number on this page comes from one of two public runs of benchmark workflows in the engine repository: the first part times bar backtests and parameter grids, the second market making on order-book data. Both install each engine from PyPI on a GitHub-hosted runner, the same wheel a user installs, then compare results before they compare speed. At 10,000,000 bars one backtest with the full metric suite costs 210.7 ms, 391x faster than vectorbt and 5.8x faster than raptorbt. On a grid of 10,000 combinations at 200,000 bars, the gap over raptorbt widens to 36.2x, because only one of these engines has a sweep API.

This page is the whole result, not the flattering half: the workloads where the engines diverge, the grids a challenger could not hold in memory, and the points too noisy to quote are all here. Nothing below has to be taken on trust: the run that produced it is public, log and job summary included, and so is the harness that measured it: workload definitions, parity checks, and the method it all rests on. Market making and HFT has its own run and its own section, further down. The GPU timings come from a workflow of their own, which runs the same parameter grids on a card rented from Modal, since GitHub's runners have none: they are right after the sweep table, where 150,000 combinations at 200,000 bars run 25.6x faster on the card than on the runner.

Where the bar-engine numbers came from

Measured
2026-11-08
Commit
05d1ca42e9f1
backtests, cold start, memory
Machine
AMD EPYC 9V74 80-Core Processor, 4 logical cores, 16.8 GB RAM
OS
Linux 7.0.0-1019-azure x86_64
Python
3.12.16
Engines
manifoldbt 0.33.0, vectorbt 0.28.4, raptorbt 0.9.0, nautilus_trader 1.231.0, numpy 2.4.3, numba 0.64.0, pandas 2.3.3
parameter sweeps
Machine
AMD EPYC 9V74 80-Core Processor, 4 logical cores, 16.8 GB RAM
OS
Linux 7.0.0-1019-azure x86_64
Python
3.12.16
Engines
manifoldbt 0.33.0, vectorbt 0.28.4, raptorbt 0.9.0, nautilus_trader 1.231.0, numpy 2.4.3, numba 0.64.0, pandas 2.3.3

A GitHub-hosted runner is a small shared machine, which is the point: it is the floor rather than the ceiling, and anyone can rent the identical one for free. Absolute times on a workstation are lower; the ratios are what travel.

One backtest, start to finish

Median wall time of a single call, the engines interleaved over 7 repetitions so a slow moment on the runner hits all of them equally. Data loading is excluded and the warmup call is discarded, so this is the simulation itself. Every row below produced identical results across the engines: a row where they disagree is never timed here, it is listed further down instead.

Single backtest median wall time by engine, workload and series length, with each challenger's time against the reference
WorkloadBarsmanifoldbtvectorbtraptorbtNautilusTrader
sma_cross100,0004.3 ms139.8 ms32.5x slower ~7.7 ms1.8x slower ~4.20 s993x slower ~
sma_cross_metrics100,0004.3 ms700.8 ms159x slower ~7.6 ms1.8x slower ~4.22 s974x slower ~
sma_cross1,000,00022.8 ms1.65 s73.5x slower80.0 ms3.5x slowernot run> 100,000 bars
sma_cross_metrics1,000,00024.2 ms6.70 s277x slower93.1 ms3.9x slowernot run> 100,000 bars
sma_cross10,000,000197.8 ms20.68 s104x slower1.22 s6.1x slowernot run> 100,000 bars
sma_cross_metrics10,000,000210.7 ms82.22 s391x slower1.22 s5.8x slowernot run> 100,000 bars
ema_rsi_fees100,0005.9 ms147.8 ms24.9x slowernot run9.07 s1,543x slower
ema_rsi_fees1,000,00036.5 ms1.93 s51.1x slowernot runnot run> 100,000 bars
sma_cross_costs100,0004.0 ms142.2 ms34.8x slower ~not run4.21 s1,048x slower ~
sma_cross_costs1,000,00024.7 ms1.96 s74.1x slower ~not runnot run> 100,000 bars
sma_cross_costs10,000,000202.6 ms21.39 s106x slowernot runnot run> 100,000 bars
multi_asset100,00015.4 ms171.8 ms11.1x slowernot run38.22 s2,464x slower
multi_asset1,000,000130.9 ms2.05 s15.7x slower ~not runnot run> 100,000 bars

Under each challenger's time: how much slower or faster it is than manifoldbt, as the median of the per-repetition ratios, not a ratio of medians. A ~ marks a point whose interquartile range exceeded 15% of its median: indicative only.

The parameter sweep that validates a strategy

One backtest tells you almost nothing; the grid around it is the actual job. Manifold-BT runs a grid as one native fan-out call, with the data loaded once and the combinations spread over every core. The other three engines have no sweep API, so a grid there is a Python loop paying a full backtest per cell. Wall time for the whole grid, with each challenger's time against manifoldbt under it:

Parameter sweep wall time by engine, grid size and series length, with each challenger's time against the reference
BarsCombinationsmanifoldbtvectorbtraptorbtNautilusTrader
20,0005,000236.1 ms4.12 s17.4x slower5.15 s21.8x slowernot run
20,00020,000900.0 msnot run20.27 s22.5x slowernot run
200,00010,0003.53 snot run127.81 s36.2x slowernot run
200,000150,00072.99 snot runnot runnot run

Peak memory each engine adds running the grid

Peak memory added by each engine running a parameter grid once, by grid size and series length
BarsCombinationsmanifoldbtvectorbtraptorbtNautilusTrader
20,0005,00038 MB2.5 GB37 MB-
20,00020,00080 MB-63 MB-
200,00010,000345 MB-370 MB-
200,000150,000834 MB---

Every grid is checked cell by cell before any of it is timed: the engines are joined on the parameter pair they actually ran, not on position, and the worst disagreement in the grid is what the gate sees. 2 repetitions per point. RAM is probed in a separate single-engine process, because interleaving is what makes a memory reading worthless: a dash there means the probe did not run for that engine at that point, not that it used nothing.

Where a challenger was not run

An empty cell in a speed table reads as a defeat, so the reason is written out instead.

  • NautilusTrader, 5,000 combinations at 20,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 0.96 s per backtest at 20,000 bars (mean over 16 cells spread across this grid), which puts this grid of 5,000 combinations at about 80 minutes per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.
  • vectorbt, 20,000 combinations at 20,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file.
  • NautilusTrader, 20,000 combinations at 20,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 1.12 s per backtest at 20,000 bars (mean over 16 cells spread across this grid), which puts this grid of 20,000 combinations at about 6.2 hours per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.
  • vectorbt, 10,000 combinations at 200,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file.
  • NautilusTrader, 10,000 combinations at 200,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 10.23 s per backtest at 200,000 bars (mean over 16 cells spread across this grid), which puts this grid of 10,000 combinations at about 28.4 hours per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.
  • vectorbt, 150,000 combinations at 200,000 bars. vectorbt materialises the simulation per combination (measured: 1.57 MB per combination at 20k bars), so this grid would need tens of gigabytes. Running it would measure the swap file.
  • raptorbt, 150,000 combinations at 200,000 bars. raptorbt has no fan-out API for a parameter grid on one instrument, so a grid is a Python loop of full backtests. Measured: 11.9 ms per backtest at 200,000 bars (mean over 16 cells spread across this grid), which puts this grid of 150,000 combinations at about 30 minutes per call. That is past the time budget of the job. It is not run.
  • NautilusTrader, 150,000 combinations at 200,000 bars. NautilusTrader has no parameter-grid API, so a grid is a Python loop of full backtests, each one replaying every bar through the simulated exchange. Measured: 18.68 s per backtest at 200,000 bars (mean over 16 cells spread across this grid), which puts this grid of 150,000 combinations at about 778.2 hours per call. Looped grids are held to 64 combinations, so that a point stays inside the time budget of the job. It is not run.

The same sweeps on a GPU

GitHub's runners have no GPU, so the grids above are timed on the CPU path only. A second workflow runs the same grids, on the same generated bars and with the same strategy, on an NVIDIA L4 rented from Modal, with the wheel installed from PyPI as everywhere else on this page. It times the card alone: the CPU column below is the runner's time from the table above, for the same grid and the same data. GPU acceleration is a Researcher feature.

Measured
2026-11-08
Card
NVIDIA L4, 23034 MiB, driver 610.57.04
Engine
manifoldbt 0.33.0
Parameter sweep wall time on a GPU, next to the CPU time of the same grid on the benchmark runner, with the cells identical to the CPU result
BarsCombinationsGPUCPU (runner)Card vs runnerBit-identical
20,0005,00031.8 msfirst call 455.1 ms236.1 ms7.4x faster5,000 / 5,000
20,00020,000109.4 msfirst call 119.8 ms900.0 ms8.2x faster20,000 / 20,000
200,00010,000229.6 msfirst call 240.3 ms3.53 s15.4x faster10,000 / 10,000
200,000150,0002.85 sfirst call 2.99 s72.99 s25.6x faster150,000 / 150,000

GPU: medians of 7 repetitions, after the card is calibrated and each grid has run once untimed, since that first call compiles the kernels; it is shown under each time because it is what a first sweep costs. Every cell of the GPU grid is compared with the same grid run once on the CPU of the GPU machine, keyed by parameter pair, and the last column counts the cells equal to the bit. The ratio crosses two machines, a rented NVIDIA L4 and a four-vCPU GitHub runner: it says what the sweep costs on the runner the CPU numbers come from against what it costs on the card, not how the card compares with the CPU beside it. A ratio is only given when both runs measured the same version on the same data. A ~ marks a time whose interquartile range exceeded 15% of its median: indicative only.

The run is reproducible like the others: fork the repository, add a Modal token and a licence as secrets, and run Benchmark GPU sweeps (Modal), or run the harness on any machine with an NVIDIA card. Method and code: benchmarks/gpu.

Not run, and why

  • NautilusTrader on sma_cross, sma_cross_metrics, ema_rsi_fees, sma_cross_costs, multi_asset, bracket_sl_tp. Not run above 100,000 bars. NautilusTrader is event-driven: every bar goes through a simulated exchange and a Python strategy callback, so one call costs seconds per 100,000 bars and grows linearly with the series. Repeated and interleaved, a longer series does not fit the time budget of the job. The measured time at the ceiling is in the table, and bench.py --engine-max-bars nautilus_trader=N lifts the ceiling for a local run.
  • raptorbt on ema_rsi_fees. Two blockers, either one sufficient. Sizing: raptorbt has no fixed-quantity mode. position_sizes is a fraction of equity (measured: 0.5 buys exactly half the equity of the bar before the entry), lot_size rounds a computed size down to a multiple, and alloted_capital fixes the notional, not the quantity. Reproducing units=5 would mean feeding a fraction derived from an equity curve that does not exist until the run is over. Indicator: raptorbt.ema seeds on a simple mean of the first period bars and emits from bar period-1, while manifoldbt seeds on the first observation and emits from bar 0 (measured: 11 leading NaN for span 12, and ema[11] equal to sma(12)[11] to the last bit). The two are the same recursion with a different warmup, so the signal differs early and the round-trip count with it. Its sma and rsi do match, which is why the other three workloads run.
  • raptorbt on sma_cross_costs. Same blocker as the other fixed-quantity workload: raptorbt has no units sizing, and a cost model on top of a fraction of equity compares policy rather than arithmetic. See ema_rsi_fees for the measurements behind that.
  • raptorbt on multi_asset. No multi-instrument portfolio on the entry point this harness drives: run_single_backtest is one instrument, and run_multi_backtest broadcasts over strategies rather than over assets. 0.9.0 does add run_portfolio_backtest, but its allocation model is a different one and would need its own parity work before any timing from it could be published. The units blocker applies here too.

Cold start and memory

The two costs a timing loop hides. Cold start is what a fresh interpreter pays before the first result exists, which is what an agent, a CI job or a one-shot script actually feels. Memory is measured in a dedicated single-engine process, never in the interleaved timing run, where the peak sampled during one engine's call would include every other engine's allocations.

Cold start and memory footprint by engine
manifoldbtvectorbtraptorbtNautilusTrader
Fresh process to first backtest0.41 s4.78 s0.31 s1.54 s
Minus the 0.29 s bare interpreter0.12 s4.49 s0.03 s1.25 s
RAM added by the run, per million barsunder 0.1 MB139 MB5.2 MB6.7 MBat 100,000 bars

Cold start is timed on sma_cross at 20,000 bars. Memory is measured on 2,000,000 bars. An engine held to a ceiling is measured at it and scaled to a million bars like the others; its cell says at how many.

What the engines agreed on

A speed number is worth nothing if the engines computed different things, so every workload is compared before it is timed: final equity against starting capital, round-trip count, fees. 13 of the 16 measured points came back exact. 3 carry a known, documented divergence and are kept out of the headline. None disagreed unexpectedly.

Workloads where the engines diverge for a documented reason
WorkloadBarsmanifoldbtvectorbtraptorbtNautilusTrader
bracket_sl_tp100,0007.2 ms142.9 ms6.3 ms9.54 s
bracket_sl_tp1,000,00051.7 ms2.45 s61.9 msnot run
bracket_sl_tp10,000,000445.8 ms24.22 s1.26 snot run
  • vectorbt. Re-entry on the exit bar. When a bracket fires intrabar and the entry condition still holds at that bar's close, manifoldbt books two orders on that bar (the stop or target exit, then a fresh entry at the close); vectorbt processes one order per bar and re-enters on the next bar instead. Neither is wrong, and on controlled bars the bracket fills themselves match exactly (see the cross-engine parity suite shipped with the library). The harness counts the affected round-trips so the size of the divergence is measured, not asserted.
  • raptorbt. Same fork in the road, taken further: raptorbt does not re-arm at all. Once a bracket closes a position, the entry level being still true is not enough to open another one; it waits for the level to go false and true again. So on the same bars the three engines book a different number of round-trips from the same signal, manifoldbt re-entering on the exit bar, vectorbt on the next bar, raptorbt not until the next crossing. The count of affected round-trips is measured rather than asserted, and it is the same population for both challengers.
  • NautilusTrader. Re-entry sizing. Entries, exits and fill prices are the same trades as the reference's, round-trip for round-trip: the stop and the target are native contingent orders, they fill at their own levels, and because the simulated exchange runs before the strategy sees the bar, a position stopped out intrabar is re-entered at that bar's close just as the reference does. What differs is the quantity of that re-entry. The reference sizes it from the equity at the close, counting the position it has just closed at the close price, so it buys back the quantity it held; Nautilus sizes it from the cash balance after the stop or target has been booked at its own price. Neither is wrong, the two quantities differ by the gap between the exit price and the close, and that gap is carried into every later trade. The harness measures it (re-entries on the exit bar, and how far each one's quantity sits from the position closed on that bar) instead of asserting it.
  • On bracket_sl_tp at 100,000 bars, 922 of 2,303 manifoldbt round-trips re-enter on the exit bar (40%), from 630 stop and 387 target exits. raptorbt books 1,381. NautilusTrader books 2,303. Of those, 922 re-enter on the exit bar, with a quantity that sits a median of 2.7e-4 (at most 1.1e-3), in relative terms, away from the position closed on that bar.

Run it yourself

A self-reported benchmark is worth what its reproduction costs. Fork the repository and press "Run workflow" on Benchmark vs vectorbt, raptorbt and NautilusTrader to get this table on your own runner, or run the harness locally:

reproduce.sh
# Every engine comes from PyPI, exactly as you would install it.
git clone https://github.com/manifoldbt/manifoldbt
cd manifoldbt/benchmarks/vs_vectorbt
pip install manifoldbt
pip install -r requirements-lock.txt

# Backtests, cold start and memory
python bench.py --bars 100000 1000000 10000000 --reps 7 \
  --cold-start-reps 3 --memory-bars 2000000

# The parameter grids, which need a licence (see below)
python bench.py --workloads sma_cross --bars 100000 \
  --sweep 20000:5000 20000:20000:oos=vectorbt 200000:10000:oos=vectorbt 200000:150000:oos=vectorbt,raptorbt --sweep-reps 2

The sweep points need a licence: an unlicensed fan-out call waits out a fixed interval before any work starts, so the stopwatch would be timing the wait rather than the engine. Everything else runs on the free tier.

Market making and HFT, engine by engine

A second public benchmark, on order-book data: manifoldbt, hftbacktest and NautilusTrader run the same trading rules, written once, on one pinned day of public market data. The harness checks that the engines did the same work first, and only then reports how long each took. With market making, counting fills is not enough to know whether they did: an engine that serves an order in several smaller pieces shows more fills for the same executed quantity, and an engine that never serves a print that went through a quote shows fewer for the same market.

Parity is therefore read in three layers: the actions (which quotes were sent, at which wake-up, on which side, at which price), the verdicts (for a quote both engines sent, was it served, and for how much) and the totals (fills, executed quantity). manifoldbt is the reference: each challenger is compared with it, never with the other challenger, and a time beside a name is that engine's time against manifoldbt's, as a ratio of medians. A ~ marks a median whose interquartile range exceeded 15%: indicative only. Where the engines differ, the mechanisms are listed under why the engines differ.

Where the market-making numbers came from

Measured
2026-11-08
Commit
05d1ca42e9f1
Market data day
2025-08-13, 3 interleaved repetitions
data, pinned by SHA-256
Order book archive (Bybit spot BTCUSDT, 200 levels)
https://quote-saver.bycsi.com/orderbook/spot/BTCUSDT/2025-08-13_BTCUSDT_ob200.data.zip, 89.9 MB
63e1f91c9b3e28fd0189a0bf1773eadf0e02cc0d63c1608d4403013da848bc38
Trade archive (Bybit spot BTCUSDT)
https://public.bybit.com/spot/BTCUSDT/BTCUSDT_2025-08-13.csv.gz, 5.1 MB
5a9277a3c73f93af1811917a29cf723fc34b5d4f1a17868dbd90d5fa26dd671c
Prepared arrays both engines read
70daffab0e402e39de58db80aa28bcbcef8592a1eaea5eba4033bcc0386c3759
Nasdaq TotalView-ITCH extract, AAPL
AAPL.itch, 40.3 MB
6d66ec7f49fc61971f4617208d179258ebf26a976a29a10255dfb7574fd07106

The run downloads these from the source and stops if a fingerprint changed. Nothing is redistributed. Two runs that print the same fingerprints read the same bytes.

books and best bid/ask
Machine
4 logical CPUs (2 physical), 16.8 GB RAM
OS
Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 x86_64
Python
3.12.16
Engines
manifoldbt 0.33.0, hftbacktest 2.4.4, nautilus_trader 1.231.0, numba 0.68.0, numpy 2.2.6, pandas 3.1.0, pyarrow 26.0.0
manifoldbt library SHA-256
84d4cb78e690b29b3d6e41e15985b8071eee7d8680a8aa9e293faa63edae1de1
order by order
Machine
4 logical CPUs (2 physical), 16.8 GB RAM
OS
Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 x86_64
Python
3.12.16
Engines
manifoldbt 0.33.0, hftbacktest 2.4.4, nautilus_trader 1.231.0, numba 0.68.0, numpy 2.2.6, pandas 3.1.0, pyarrow 26.0.0
manifoldbt library SHA-256
84d4cb78e690b29b3d6e41e15985b8071eee7d8680a8aa9e293faa63edae1de1

Order books by price

Bybit spot BTCUSDT, one pinned day. The best bid/ask and the top 25 levels are cut from the same replayed states.

mm_ladder_l2

Two post-only quotes every 100 ms, one level further back per clip of inventory held, inventory capped at 5 clips. Reads 25 levels a side and the trades.

25 levels a side + trades · 5 / 5 / 5 / 2 ms

mm_ladder_l2: median wall time by engine, with the time against manifoldbt, replay-only time, RAM added by the engine and first-run time
EngineStrategy runReplay onlyAdded RAMFirst run in a process
manifoldbt541.0 ms178.0 ms570 MB936.0 ms
hftbacktest1.53 s2.8x slower384.0 ms1.6 GB6.31 s
NautilusTrader133.25 s246x slower42.42 s1.4 GB133.25 s
  • NautilusTrader: 124.08 s in the engine, 7.89 s building its data objects; both are inside the strategy run.
  • Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 20.0 ms, hftbacktest 1.91 s.
  • Actions part at wake-up 310, verdicts differhftbacktest against manifoldbt
    • hftbacktest sent 133,194 quotes and manifoldbt 133,494; 124,951 of the 130,505 they share are at the same price.
    • Verdicts on the shared quotes: served by both 27,222, only by hftbacktest 6, only by manifoldbt 49; the same executed quantity on 27,073 of the 27,222 served by both.
    • Quotes served: 28,897 (hftbacktest), 28,766 (manifoldbt). Fills booked: 28,897 against 36,579; a fill is a piece of an order, so this is not a count of quotes served.
  • Actions part at wake-up 44, verdicts differNautilusTrader against manifoldbt
    • NautilusTrader sent 116,412 quotes and manifoldbt 133,494; 72,848 of the 102,871 they share are at the same price.
    • Verdicts on the shared quotes: served by both 18,361, only by NautilusTrader 95, only by manifoldbt 293; the same executed quantity on 18,104 of the 18,361 served by both.
    • Quotes served: 25,663 (NautilusTrader), 28,766 (manifoldbt). Fills booked: 25,663 against 36,579; a fill is a piece of an order, so this is not a count of quotes served.

mm_touch_bbo

The same maker on a best bid/ask feed: it can only join the touch.

best bid/ask + trades · 5 / 5 / 5 / 2 ms

mm_touch_bbo: median wall time by engine, with the time against manifoldbt, replay-only time, RAM added by the engine and first-run time
EngineStrategy runReplay onlyAdded RAMFirst run in a process
manifoldbt376.0 ms144.0 ms484 MB597.0 ms
hftbacktest405.0 ms1.1x slower64.0 ms369 MB5.24 s
NautilusTrader62.07 s165x slower7.36 s1011 MB62.07 s
  • NautilusTrader: 59.54 s in the engine, 1.33 s building its data objects; both are inside the strategy run.
  • Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 19.0 ms, hftbacktest 844.0 ms.
  • Actions part at wake-up 488, verdicts differhftbacktest against manifoldbt
    • hftbacktest sent 76,682 quotes and manifoldbt 76,847; 75,198 of the 75,198 they share are at the same price.
    • Verdicts on the shared quotes: served by both 30,102, only by hftbacktest 17, only by manifoldbt 24; the same executed quantity on 29,966 of the 30,102 served by both.
    • Quotes served: 31,023 (hftbacktest), 30,731 (manifoldbt). Fills booked: 31,023 against 43,664; a fill is a piece of an order, so this is not a count of quotes served.
  • Actions part at wake-up 37, verdicts differNautilusTrader against manifoldbt
    • NautilusTrader sent 68,913 quotes and manifoldbt 76,847; 63,539 of the 63,539 they share are at the same price.
    • Verdicts on the shared quotes: served by both 25,123, only by NautilusTrader 998, only by manifoldbt 631; the same executed quantity on 25,045 of the 25,123 served by both.
    • Quotes served: 29,502 (NautilusTrader), 30,731 (manifoldbt). Fills booked: 29,502 against 43,664; a fill is a piece of an order, so this is not a count of quotes served.

flow_taker_bbo

Follow the aggressor imbalance of the last 34 s (arm at +/-0.9), market orders, flip on the opposite crossing. Trade clock.

best bid/ask + trades · 20 ms order delay

flow_taker_bbo: median wall time by engine, with the time against manifoldbt, replay-only time, RAM added by the engine and first-run time
EngineStrategy runAdded RAMFirst run in a process
manifoldbt87.0 ms366 MB305.0 ms
hftbacktest83.0 ms1.0x faster329 MB3.10 s
NautilusTrader9.83 s113x slower646 MB9.83 s
  • NautilusTrader: 7.51 s in the engine, 1.32 s building its data objects; both are inside the strategy run.
  • Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 20.0 ms, hftbacktest 849.0 ms.
  • Same actionshftbacktest against manifoldbt
    • 364 decisions at the same times and on the same sides; 364 orders identical.
    • Fills 364 against 364; executed quantity 7.27 against 7.27; notional -0.034 bp from manifoldbt.
  • Same actionsNautilusTrader against manifoldbt
    • 364 decisions at the same times and on the same sides; 364 orders identical.
    • Fills 447 against 364; executed quantity 7.27 against 7.27; notional +0.006 bp from manifoldbt.

The open-loop probe

Not a strategy but an instrument. Its quotes depend on the market only, so every quote gets a verdict (served or not) that does not depend on what the engine did before. That isolates the queue and fill model of each engine.

probe_l2_zero

Open-loop probe: one post-only quote a side at the touch, cancelled and posted again at every wake-up, no inventory, zero latency.

25 levels a side + trades · zero latency · window 10:00 to 11:00 UTC

probe_l2_zero: median wall time by engine, with the time against manifoldbt, replay-only time, RAM added by the engine and first-run time
EngineStrategy runAdded RAMFirst run in a process
manifoldbt257.0 ms374 MB523.0 ms
hftbacktest86.0 ms3.0x faster373 MB4.91 s
NautilusTrader19.03 s74.1x slower933 MB19.03 s
  • NautilusTrader: 17.65 s in the engine, 348.0 ms building its data objects; both are inside the strategy run.
  • Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 20.0 ms, hftbacktest 674.0 ms.
  • Same actions, verdicts differhftbacktest against manifoldbt
    • The same 71,994 quotes, at the same prices.
    • Verdicts on the shared quotes: served by both 1,590, only by hftbacktest 0, only by manifoldbt 16; the same executed quantity on 1,561 of the 1,590 served by both.
    • Quotes served: 1,590 (hftbacktest), 1,606 (manifoldbt). Fills booked: 1,590 against 1,792; a fill is a piece of an order, so this is not a count of quotes served.
  • Actions differ on 4 of 71,998 quotes, verdicts differNautilusTrader against manifoldbt
    • NautilusTrader sent 71,998 quotes and manifoldbt 71,994; 71,994 of the 71,994 they share are at the same price.
    • Verdicts on the shared quotes: served by both 1,311, only by NautilusTrader 0, only by manifoldbt 295; the same executed quantity on 1,249 of the 1,311 served by both.
    • Quotes served: 1,311 (NautilusTrader), 1,606 (manifoldbt). Fills booked: 1,311 against 1,792; a fill is a piece of an order, so this is not a count of quotes served.

probe_l2_nominal

The same open-loop probe under the 5 / 5 / 5 / 2 ms venue.

25 levels a side + trades · 5 / 5 / 5 / 2 ms · window 10:00 to 11:00 UTC

probe_l2_nominal: median wall time by engine, with the time against manifoldbt, replay-only time, RAM added by the engine and first-run time
EngineStrategy runAdded RAMFirst run in a process
manifoldbt256.0 ms374 MB524.0 ms
hftbacktest86.0 ms3.0x faster373 MB4.85 s
NautilusTrader20.02 s78.2x slower935 MB20.02 s
  • NautilusTrader: 18.65 s in the engine, 348.0 ms building its data objects; both are inside the strategy run.
  • Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 18.0 ms, hftbacktest 684.0 ms.
  • Actions differ on 2 of 71,996 quotes, verdicts differhftbacktest against manifoldbt
    • hftbacktest sent 71,996 quotes and manifoldbt 71,994; 71,994 of the 71,994 they share are at the same price.
    • Verdicts on the shared quotes: served by both 1,430, only by hftbacktest 1, only by manifoldbt 8; the same executed quantity on 1,408 of the 1,430 served by both.
    • Quotes served: 1,431 (hftbacktest), 1,438 (manifoldbt). Fills booked: 1,431 against 1,567; a fill is a piece of an order, so this is not a count of quotes served.
  • Actions differ on 4 of 71,998 quotes, verdicts differNautilusTrader against manifoldbt
    • NautilusTrader sent 71,998 quotes and manifoldbt 71,994; 71,994 of the 71,994 they share are at the same price.
    • Verdicts on the shared quotes: served by both 1,132, only by NautilusTrader 5, only by manifoldbt 306; the same executed quantity on 1,111 of the 1,132 served by both.
    • Quotes served: 1,137 (NautilusTrader), 1,438 (manifoldbt). Fills booked: 1,137 against 1,567; a fill is a piece of an order, so this is not a count of quotes served.

Order by order

Nasdaq TotalView-ITCH: every order is visible, so the queue position is known rather than modelled.

mbo_join_mbo

Join the best bid and ask with a round lot, stand a side aside behind a crowded best or at the inventory bound. Nasdaq TotalView-ITCH, order by order (AAPL).

order by order · zero latency

mbo_join_mbo: median wall time by engine, with the time against manifoldbt, replay-only time, RAM added by the engine and first-run time
EngineStrategy runReplay onlyAdded RAMFirst run in a process
manifoldbt488.0 ms326.0 ms699 MB730.0 ms
hftbacktest635.0 ms1.3x slower610.0 ms262 MB5.56 s
NautilusTrader26.60 s54.5x slower11.49 s737 MB26.60 s
  • NautilusTrader: 24.18 s in the engine, 1.78 s building its data objects; both are inside the strategy run.
  • Preparing each engine's input (an event array, a store opened) is timed apart and is not in the run: manifoldbt 26.0 ms, hftbacktest 140.0 ms.
  • Actions part at wake-up 428, verdicts differhftbacktest against manifoldbt
    • hftbacktest sent 20,124 quotes and manifoldbt 20,002; 19,215 of the 19,215 they share are at the same price.
    • Verdicts on the shared quotes: served by both 4,790, only by hftbacktest 83, only by manifoldbt 174; the same executed quantity on 3,638 of the 4,790 served by both.
    • Quotes served: 5,162 (hftbacktest), 5,170 (manifoldbt). Fills booked: 5,162 against 10,261; a fill is a piece of an order, so this is not a count of quotes served.
  • Actions part at wake-up 53, verdicts differNautilusTrader against manifoldbt
    • NautilusTrader sent 20,067 quotes and manifoldbt 20,002; 19,038 of the 19,038 they share are at the same price.
    • Verdicts on the shared quotes: served by both 4,891, only by NautilusTrader 1,499, only by manifoldbt 0; the same executed quantity on 3,854 of the 4,891 served by both.
    • Quotes served: 6,828 (NautilusTrader), 5,170 (manifoldbt). Fills booked: 6,828 against 10,261; a fill is a piece of an order, so this is not a count of quotes served.

The probe, quote by quote

Every quote of the probe is judged on its own: served or not. The counts are over the quotes both engines sent at the same price, so a difference in them is a difference in the queue and fill model and nothing else.

Open-loop probe: how many quotes each engine served, against manifoldbt, by latency setting
Engine, latencyServed by bothOnly the engineOnly manifoldbtNeither
hftbacktestzero latency1,59001670,388
NautilusTraderzero latency1,311029570,388
hftbacktest5 / 5 / 5 / 2 ms1,4301870,555
NautilusTrader5 / 5 / 5 / 2 ms1,132530670,551

Parameter sweeps

manifoldbt sweeps a quoting strategy through run_sweep (every combination keeps its order journal) and run_sweep_lite (metrics only): the book and the tape are loaded once and the combinations run in parallel inside the engine. hftbacktest and NautilusTrader have no sweep call, so what is timed is the loop of runs a user would write, and a pool of processes, each holding its own copy of the data, created and warmed beforehand and reported apart. The grids are nested: each contains the smaller ones.

Maker: order size x inventory bound x levels back on each side

window 10:00 to 12:00 UTC

combinations per second against grid size, maker family
  • manifoldbt run_sweep_lite
  • hftbacktest, 4 processes
  • NautilusTrader, 4 processes
  • dashed and hollow: grid time projected
0.1/s1/s10/s100/s41664256combinations in the gridmanifoldbt run_sweep_lite, 4 combinations: 33.9/smanifoldbt run_sweep_lite, 16 combinations: 57.6/smanifoldbt run_sweep_lite, 64 combinations: 68.8/smanifoldbt run_sweep_lite, 256 combinations: 65.6/s65.6/shftbacktest, pool of 4, 4 combinations: 15.8/shftbacktest, pool of 4, 16 combinations: 22.6/shftbacktest, pool of 4, 64 combinations: 23.7/shftbacktest, pool of 4, 256 combinations: 24.5/s24.5/sNautilusTrader, pool of 4, 4 combinations: 0.248/sNautilusTrader, pool of 4, 16 combinations: 0.250/s (measured on 8 of them, the grid time is projected)NautilusTrader, pool of 4, 64 combinations: 0.254/s (measured on 8 of them, the grid time is projected)NautilusTrader, pool of 4, 256 combinations: 0.256/s (measured on 8 of them, the grid time is projected)0.26/s
Throughput of the same grid in each engine, both axes logarithmic. A line that stays level is an engine that cost the same per combination whatever the grid; one that climbs is paying a fixed cost once and spreading it.
Maker: order size x inventory bound x levels back on each side: combinations per second by call and grid size, with the wall time and the peak memory of the call under it
Combinations in the grid41664256
manifoldbt run_sweep_lite33.9/s118.0 ms305 MB57.6/s278.0 ms315 MB68.8/s930.0 ms313 MB65.6/s3.90 s314 MB
manifoldbt run_sweep30.5/s131.0 ms323 MB50.0/s320.0 ms436 MB58.4/s1.09 s736 MB57.1/s4.49 s2.2 GB
hftbacktest, 4 processes15.8/s253.8 ms22.6/s707.0 ms23.7/s2.70 s24.5/s10.43 s
hftbacktest, one process9.6/s416.7 ms9.8/s1.63 s9.8/s6.56 s9.5/s26.91 s
NautilusTrader, 4 processes0.248/s16.13 s0.250/s64.1 s projected0.254/s4.2 min projected0.256/s16.7 min projected
NautilusTrader, one process0.114/s35.15 s0.118/s2.3 min projected0.120/s8.9 min projected0.120/s35.6 min projected
  • At the rate measured on the largest grid, one hour of this machine covers about 236,185 with manifoldbt run_sweep_lite, 88,321 with hftbacktest, 921 with NautilusTrader.
  • Every pool of processes returned exactly what the loop of runs returned, combination by combination.
  • NautilusTrader ran at most 8 combinations of a grid: its rate is measured on those and the time for the whole grid is projected from it, which is what the dashed segments and the "projected" cells say.
  • manifoldbt: 28 of 28 sampled combinations came out of the sweep identical, to the bit, to the same combination run alone.

Order delay as an axis: delay x inventory bound x levels back

window 10:00 to 12:00 UTC

Order delay as an axis: delay x inventory bound x levels back: combinations per second by call and grid size, with the wall time and the peak memory of the call under it
Combinations in the grid16
manifoldbt run_sweep_lite57.3/s279.0 ms315 MB
manifoldbt run_sweep49.8/s321.0 ms453 MB
hftbacktest, 4 processes23.3/s686.4 ms
hftbacktest, one process9.8/s1.63 s
NautilusTrader, 4 processes0.255/s62.8 s projected
NautilusTrader, one process0.116/s2.3 min projected
  • Every pool of processes returned exactly what the loop of runs returned, combination by combination.
  • NautilusTrader ran at most 8 combinations of a grid: its rate is measured on those and the time for the whole grid is projected from it, which is what the dashed segments and the "projected" cells say.
  • manifoldbt: 8 of 8 sampled combinations came out of the sweep identical, to the bit, to the same combination run alone.

Flow taker on the trade clock: flow threshold x order size

Flow taker on the trade clock: flow threshold x order size: combinations per second by call and grid size, with the wall time and the peak memory of the call under it
Combinations in the grid12
manifoldbt run_sweep24.4/s492.0 ms823 MB
hftbacktest, 4 processes22.6/s532.1 ms
hftbacktest, one process12.2/s985.2 ms
NautilusTrader, 4 processes0.184/s65.1 s projected
NautilusTrader, one process0.083/s2.4 min projected
  • Every pool of processes returned exactly what the loop of runs returned, combination by combination.
  • NautilusTrader ran at most 6 combinations of a grid: its rate is measured on those and the time for the whole grid is projected from it, which is what the dashed segments and the "projected" cells say.
  • manifoldbt: 4 of 4 sampled combinations came out of the sweep identical, to the bit, to the same combination run alone.

Why the engines differ

The three engines are set to the same venue wherever they can express it. Under the nominal venue an order, a cancel and a response each take 5 ms and the feed takes 2 ms; fees are 0 for a maker and 4 basis points for a taker; the wake-ups fall on the same instants; and each engine runs the closest equivalent of the others' queue model, listed in the harness README. hftbacktest runs with its no-partial-fill exchange, the benchmark's choice: an order is served whole at its first piece. NautilusTrader has no response delay, so the strategy emulates it. Two rules come from the engines' own semantics and are applied to all three: the wake-up clock starts at the first real book state inside the run, and the flow taker's trailing window of 34 s gives no signal until a whole window of tape has gone by.

Where they still differ, measured:

  • Partial fills. Where an order is served in several pieces, the engines that serve in pieces show more fills for the same executed quantity, and a closed-loop strategy reacts to the first piece or to the whole order accordingly. This is the first place two closed-loop runs part, and each parity line gives the wake-up.
  • NautilusTrader serves a resting order only through prints at its own price. With queue position on, a print through the level, a book that crosses the order and a shrinking of the displayed size ahead of it do not serve it. On the open-loop probe this shows as quotes that one engine serves and the other never does, and the probe table counts them.
  • On an order-by-order market the right answer can be computed. The harness ships a replay of the day in explicit first-in first-out queues (fifo_truth.py) that judges every quote the engines disagree on, engine by engine: how many it served although an exact queue had not served them yet, and how many it missed. It is a separate tool, and its verdicts are not part of the run published above.
  • Market orders. The taker's decisions and orders are compared exactly. The fill price is the price of the first print at or after the order's arrival in manifoldbt's trade-clock engine, and the book at the arrival in the other two; the measured gap of the executed notional is in the parity line.

The scenarios that reproduce each case on a book of a few lines, for all three engines, are in the harness README.

Run it yourself

The same commands run in GitHub Actions on public runners (bench-vs-hft.yml), so every number above has a run behind it. No account and no API key is needed for the data.

reproduce-market-making.sh
# Every engine comes from PyPI, exactly as you would install it.
git clone https://github.com/manifoldbt/manifoldbt
cd manifoldbt/benchmarks/vs_hft
pip install manifoldbt
pip install -r requirements-lock.txt

# One pinned day of public data, checked against its SHA-256
python prep_cli.py 2025-08-13

# Order books by price: the makers, the taker and the probe
python bench.py --day 2025-08-13 --reps 3 \
  --workloads mm_ladder_l2 mm_touch_bbo flow_taker_bbo probe_l2_zero probe_l2_nominal \
  --out results-books.json

# Order by order: the Nasdaq day file is read once and AAPL is kept
python itch_cli.py --day 2025-11-28 --symbols AAPL --out itch-extract
python bench.py --day 2025-08-13 --reps 3 \
  --itch itch-extract/AAPL.itch --workloads mbo_join_mbo \
  --out results-orders.json

# The maker sweeps
python sweep.py --family maker --day 2025-08-13 --window 36000 43200 \
  --grids 4 16 64 256 --workers 4 --out results-sweeps-maker.json
python sweep_mbt.py --family maker --day 2025-08-13 --window 36000 43200 \
  --grids 4 16 64 256 --workers 4 --identity 8 \
  --out results-sweeps-mbt-maker.json

python report.py results-books.json

Keep reading

Run your first backtest

Install Manifold-BT and reproduce the backtest above in seconds. The Rust core runs years of bars sub-second so you can sweep parameters instead of waiting.

$pip install manifoldbt