Running the benchmarks#

benchmarks/ times complete runs of a small set of representative workloads. Each release adds a few lines per case to a history file kept per hardware set, so that regressions and improvements show from one release to the next. The design is SEP 38, which replaced SEP 30’s standard suite. The rules for a performance claim and the indicative results are in Benchmarks; this page is the manual.

uv run poe bench                      # the plan and its estimated time
uv run python -m benchmarks track     # the tracking run, about an hour
uv run python -m benchmarks history   # what earlier runs recorded
uv run python -m benchmarks ab --case rdw --baseline-python OLD/bin/python

Every command uses the native CPU backend unless --backend says otherwise, and needs it installed, as a plain uv sync does. Run on a quiet machine: a dedicated workstation, or an exclusive allocation on a cluster node.

The tracking set#

python -m benchmarks list prints the plan: one line per case, stepper and width, and the estimated time from the costs each case declares (complete runs on 16 cores of an AMD EPYC 7B13, about 33 minutes for the tracking set; other machines differ). With no selection it is the tracking set:

Case

Interface

Problem

Widths

gnlse_raman_shock

prop_gnlse

Dudley–Genty–Coen supercontinuum, 0.15 m

1, production

gnlse_soliton

prop_gnlse

N = 3 soliton, one soliton period

1, production

rdw

prop_capillary

He resonant dispersive wave, four modes, 1.5 m

1, production

he_rdw_averaged

prop_capillary

the same, mode-averaged

1, production

argon_blueshift

prop_capillary

Ar ionisation blueshift, 0.2 m

1, production

argon_blueshift_averaged

prop_capillary

the same, mode-averaged

1, production

argon_blueshift_envelope

prop_capillary

the same on the envelope grid

1, production

h2_raman_averaged

prop_capillary

H2 soliton self-frequency shift, 0.4 m

1, production

ctc

prop_capillary

H2 comb-to-continuum, three modes, 0.13 m

production

radial_filament_256

prop_radial

Ar filament, 256 radial nodes, 3 cm

1, production

grin_nonlinear

prop_waveguide

four-channel graded-index fibre, 2 cm

production

kerr_beam_384

prop_free

fused-silica Kerr beam, 384², four fixed RK4IP steps

production

kerr_beam_192

prop_free

the same at 192²

1

The definitions are in benchmarks/registry.py and benchmarks/builders.py. Every other registered case (the remaining modal, free-space and advanced-guide cases, and the backend exercises) runs when named with --case, --set or --all; none is in the default selection. A case whose definition changes keeps its name and raises its revision, and the history compares rows only within one revision.

What is timed#

Each configuration (case, stepper, width) runs in its own worker process, whose thread pools are set in the environment before any scientific import. The worker:

  1. builds the case, translates it and constructs the backend, timing these separately;

  2. runs an untimed warm-up over the first 2 % of the guide, which builds FFT plans and fills caches;

  3. times --repeats complete runs (default 3). Each restarts the backend from the translated launch and goes through solprop.driver.run.run_problem, the loop of Simulation.run, with a fresh boundary monitor as Simulation.run attaches one, so it includes the plane observations, the statistics and the boundary checks: what a user’s run costs.

The result is the median of the repeats, with the minimum and maximum beside it, the accepted and rejected steps, the source evaluations, the peak resident memory, and the integrator and tolerances the run actually used. Repeats of one configuration take the same steps; the summary says so when they do not.

Library defaults. A case is the interface call a user would write. It passes no tolerance or integrator, so it runs with its interface’s defaults (SEP 37). A changed default therefore shows in the history as a new integrator or tolerance on that row. Every noisy launch fixes the same seed, so its repeats and its releases are comparable. The N = 3 soliton is the one noiseless case: a soliton-recurrence study runs without noise, and SEP 37 measured a seed multiplying its steps by 65. The Kerr beams run their declared fixed RK4IP schedule.

Steppers. --stepper takes default, multistep, multistep-interaction or rk, or a comma-separated list to compare steppers in one run:

uv run python -m benchmarks track --case rdw --case ctc --stepper default,rk

An override replaces the integrator only; the tolerances stay the interface’s. A fixed-step case admits default only, and the plan says which it skipped.

Widths. Without --threads, each case runs at its declared widths: one thread and the production width, or one of them for the most expensive cases. --threads 1,8 names the widths for every selected case. The production width is the number of physical cores in the process’s CPU allocation unless --production gives it; on a cluster, give it explicitly.

Threading. --threading sets OpenMP’s placement and waiting in every worker: passive (unpinned, idle threads sleep at once), passive-bound (the same, pinned to neighbouring cores), default (the runtime’s own short spin before sleeping), active (idle threads spin), bound (threads pinned to neighbouring cores) or bound-active. The policy is recorded with each row, and rows compare only within one policy. Binding also confines SciPy’s transforms on the host to the main thread’s core, so a free-space row’s edge reads cost more under a bound policy than under passive (the edge against 0.1.0).

On a GPU#

With a native GPU build installed (CUDA or HIP; see the native backend’s build guide), --device runs every selected case once on the device:

uv run python -m benchmarks track --device

The host still runs the production width of threads, but each row’s width is device. The hardware name is the GPU model, for example nvidia-a40 or nvidia-h200, plus the instance family on a cloud machine, and the host CPU model is recorded beside it. An AMD GPU whose rocm-smi series is the generic “AMD Radeon Graphics” is named from its PCI device or subsystem ID instead. Each row also records the device memory the run holds. The worker refuses a GPU build without --device and a CPU build with it. The case costs are CPU costs, so a device plan gives no time estimate.

Recording a run#

uv run python -m benchmarks track --append-history

The run’s bundle goes to .benchmarks/track/<date>-<hardware>/ (git ignores it): the plan, each worker’s request, environment and logs, and results.json, which is rewritten after every configuration so that an interrupted run keeps what it measured. --archive DIR copies the bundle to a directory outside the repository.

--append-history adds one row per configuration to benchmarks/history/<hardware>.csv, which is committed. Each row names its backend, and rows compare only with rows of the same backend, so the numba_cpu rows recorded at 0.1.0, before SEP 43 retired that backend, keep their own series beside the native one. It records only the whole default plan with at least three repeats, and only when:

  • every configuration finished, and its repeats took the same steps and reached the same final field to rounding, with the native backend running the threads it was given;

  • the workers imported solprop from this checkout’s src/ (run from the checkout’s own environment, uv sync, not an installed copy), with, for a native run, a native build from a commit after which nothing under native/ changed but its documentation and tests;

  • nothing under src/, native/ or benchmarks/ is uncommitted (the history files aside, so one checkout can record several runs), unless --allow-dirty, whose rows are marked dirty;

  • SOLPROP_FFTW_PLAN is unset: a row does not record how it planned, and measured planning changes its construction time and its median.

The hardware name is the CPU model and the production width, plus the instance family on a cloud machine, for example apple-m5-max-18c, amd-epyc-7763-16c or arm-neoverse-v2-16c-aws-m8g; it never contains a host name. Give a cloud machine’s instance type with --instance-type where its DMI record does not name it (AWS’s does).

python -m benchmarks history prints, per hardware and width, one line per recorded run: its date, solprop version, the configurations it ran and how many started on a busy machine, and its change against the first and the previous recorded runs. A change is the geometric mean of median-time ratios (below one is faster) over the configurations both runs share, counted in brackets. Two rows are the same configuration only with the same number of memory (NUMA) domains spanned, threading policy, case revision, stepper, resolved integrator, tolerances and noise model. A changed library default therefore starts a new series rather than reading as a speed-up, and so does an allocation spread over more memory domains than usual: on a cluster, keep the cores together (--distribution=block:block under Slurm).

Recorded hardware#

The history starts at 0.1.0. Each release reruns the same hardware sets the same way, so that their rows stay one series:

Hardware file

Machine

Width

--instance-type

apple-m5-max-18c

Apple M5 Max, macOS

18

—

intel-xeon-gold-6240-36c

2× Intel Xeon Gold 6240, whole machine (exclusive Slurm job)

36

—

amd-epyc-7763-16c

2× AMD EPYC 7763 cluster node, exclusive, 16 cores in one NUMA domain

16

—

amd-epyc-9b45-16c-gcp-c4d

Google Cloud c4d-highcpu-32, SMT off, Ubuntu 24.04

16

gcp c4d

amd-epyc-9b45-16c-gcp-c4d-windows

the same shape, Windows Server

16

gcp c4d windows

arm-neoverse-v2-16c-gcp-c4a

Google Cloud c4a-standard-16, Ubuntu 24.04

16

gcp c4a

nvidia-a40

NVIDIA A40 cluster node, CUDA build (AMPERE86)

device

—

nvidia-h200-do

cloud gpu-h200x1-141gb instance, CUDA build (HOPPER90)

device

do

amd-mi325x-do

cloud gpu-mi325x1-256gb instance, HIP build (AMD_GFX942)

device

do

  • CPU rows time the release’s wheel. A clean checkout of the release tag, its environment made with uv sync --python 3.14 --frozen --no-install-package solprop-native, then uv pip install solprop-native==<version> from PyPI, and every command run as .venv/bin/python -m benchmarks …: a later uv sync or uv run builds native in the wheel’s place. The wheel records the tag’s commit, so the history accepts it.

  • numba rows exist only at 0.1.0: a second run in the same checkout with --backend numba_cpu, on the Apple M5 Max, the Xeon Gold 6240 and the EPYC 7763. A later release cannot add them.

  • GPU rows build the tag’s native backend with the cuda or hip preset’s options into a separate environment, out of tree (uv build native --wheel --config-setting build-dir=…), so that the checkout stays clean. ROCm 10 builds the HIP backend only with -DSOLPROP_NATIVE_WARNINGS_AS_ERRORS=OFF at 0.1.0 (fixed after it).

  • The instance type separates the operating systems. The hardware name carries no operating system, so Linux and Windows on one cloud shape differ only by --instance-type. Google Cloud allowed a 32-vCPU C4D only in us-central1 at 0.1.0, so the two ran one after the other.

  • Windows ends an SSH session’s child processes with it: start the run through WMI (Invoke-CimMethod Win32_Process Create), and run one unrecorded case first, since the first worker after boot runs cold.

  • The Mac runs with Spotlight indexing paused (sudo mdutil -a -i off) and the controlling session idle. At 0.1.0 one run of gnlse_raman_shock at 18 threads took 31 s against 21 s in the recorded run and in three repeats, with the same steps: an occasional slow mode to watch for.

  • The AMD GPU name. ROCm 10 reports the cloud MI325X’s Card Series as “AMD Radeon Graphics”; the harness then names the part from its PCI IDs. The 0.1.0 rows, recorded before that fallback, had their slug and gpu_model set to it by hand; nothing else in them changed.

Comparing two builds while iterating#

ab times the same configurations with two prepared environments, for a change in progress:

uv run python -m benchmarks ab --case rdw --case argon_blueshift --threads 1 \
    --length 0.2 --baseline-python /path/to/baseline/bin/python \
    --candidate-python /path/to/candidate/bin/python

Selection works as for track. --length runs that fraction of each guide, from the entrance, for quick iteration. Each round runs every configuration once with each build, one worker each, and alternates which build goes first; --rounds sets the count (default 3) and --repeats the timed runs per worker (default 3). The harness is copied into the output directory first, so both builds run the same benchmark code.

The summary gives, per configuration, each build’s median time, the median over rounds of the candidate/baseline ratio of the per-round medians (below one is faster), the range of those ratios, whether both builds took the same steps, and the relative L2 difference of their final fields. Passing the same interpreter twice is an A/A control: its ratio range shows the noise to expect on that machine. A one-second reading before each worker labels a busy machine but never waits. ab gives no adoption verdict; the rules say how a claim is judged.