Benchmarks¶

What it costs to build a model and hand it to a solver, in specsolve and in four other ways of writing the same model. No number includes solve time. Every rival model is hand-written in bench/models/, and CI solves each against ours, so nothing here is fast because it built a different model.

dispatch

Meets a demand from a fleet of units: bounds and one balance. The shape an array library is built for, so the comparison least in specsolve's favour.

nodal

Dispatch at nodes, where a technology exists only at the nodes it is installed at: a quarter of the node and technology pairs have a variable. The chart counts the variables that exist, not the pairs.

network

A small PyPSA dispatch model: buses joined by lines, a store at every bus that carries its charge from each snapshot to the next, a technology portfolio installed at some buses only, and solar and wind availability per snapshot. Each plant draws from one of fifty pools, such as a fuel contract, and a pool caps what its plants generate together.

sector

nodal's portfolio, where each technology delivers one of five carriers, such as heat or hydrogen, at its own efficiency. A node has a balance only for the carriers its technologies deliver.

Reading the charts¶

Each panel is one model, and each line is one library. Solver chooses the solver that every panel and the table run through.

• specsolve is the thick violet line.
• The grey line is the matrix floor: a hand-written matrix handed straight to the solver's bulk API. Its distance from specsolve is what the modelling layer costs.
• The line is the median of nine rounds, and the whisker is the middle half. Overlapping whiskers are numbers this run cannot tell apart (why the median).
• A shaded funnel is a projection past the time or memory budget. It opens from the last measured point. One edge carries the last step's cost per variable forward, and the other keeps the last step's growth rate. A wide funnel is a guess the measurements do not narrow.
• ÷ specsolve divides each point by specsolve's at the same size, so specsolve is the line at 1× and a line above it is that many times slower or larger. It applies to wall time and to peak RSS. Every panel shares one window, 0× to 5× on linear axes and 0.1× to 50× on log axes, and a point outside it is named with its ratio at the edge it crosses.
• Variables narrows every panel and the table to a range of model sizes, on either axes. Drag a handle, or drag along the axis to select a new range. Each end is zero or a power of ten, and a rung counts at the nearest power. From zero, a linear axis starts at zero; from a power of ten, it starts just below the smallest size shown.
• The table divides by specsolve. Purple is where specsolve is faster, orange where the other library is, grey within ten per cent. The specsolve-in-memory and matrix floor columns are not shaded: they are references, not rivals. >30 s and >16 GB are sizes the harness refused.

A script that solves one model¶

The charts leave out what a script pays to start: the interpreter, the import and the first build. This table times a new process from launch to exit, on HiGHS.

A script that solves one small model with specsolve finishes in under half a second, under half of linopy's time (#2024).

Not measured yet¶

• Sparsity other than a quarter. nodal, network and sector are sparse at one density, a quarter of the node and technology pairs. pixi run density sweeps nodal from dense to one pair in twelve, and no published run has taken that sweep.
• Solve time. Every number stops when the model reaches the solver.

Method¶

Each measurement runs in its own process, with import excluded and teardown included, on an idle machine. Peak memory is ru_maxrss. uv run --locked bench/reproduce.py runs it again with every version pinned, and the harness has every flag and caveat, such as the allocator cost on polars' peak (#896).