Performance tuning#

A solprop run spends its time in a handful of operations repeated at every stage of every step: fast Fourier transforms between time and frequency, dense matrix products between modal amplitudes and the transverse field (for multimode and general modal guides), the pointwise nonlinear response (Kerr, plasma, Raman) at every quadrature node, and the Raman convolution. Each of these runs on a thread pool, and the pools belong to different libraries: the backend’s own kernels, FFTW, and a BLAS. This page explains which pools exist, how each is sized by default, which environment variables change them, and how to choose settings for one large run, for a scan of many runs, and on a GPU.

Two rules apply throughout:

Set threads before Python starts. Every pool below is sized when its library is loaded or when the backend is built. Some libraries allow a smaller active thread count later, but this is not a portable way to reconfigure all pools. Put the variables on the command line or in the job script, not in os.environ halfway through a notebook.
Measure your own problem. The defaults come from measurements on representative problems, but the best settings depend on the grid, the number of modes or nodes, the responses switched on and the machine. Time about ten steps of your real problem (a short fibre, n_save=2) at two or three settings before a long campaign.

None of the settings on this page changes the physics. Thread counts can change the order of floating-point sums inside the transforms, so results from different thread counts can differ by rounding (a few units in the last place); a run repeated with the same settings on the same build is reproducible.

Which backend#

The backend decides which pools exist at all (see Choosing a backend):

backend	parallel pools	notes
`numpy_ref`	NumPy’s BLAS only	the reference; single-threaded apart from matrix products, and roughly 6 to 40 times slower than threaded native
native CPU (`native`)	OpenMP kernels, FFTW (modal and spatial transforms), the Raman convolution transforms (Kokkos-FFT over FFTW, with selected macOS shapes on Accelerate’s vDSP), a vendor BLAS	uses every CPU in the process’s affinity mask unless told otherwise
native GPU (`native`, CUDA or HIP build)	the device	CPU pools are irrelevant; one device per process

backend="auto" chooses the native backend, a required dependency, and has no fallback (which backend a run uses). A locally built one (The native backend) goes a little further than the wheel, since it links a vendor BLAS and compiles for the machine’s own instruction set; which backend to use compares the backends by problem.

Which stepper#

Each interface has its own default integrator (the defaults): Multistep() on prop_capillary, prop_radial and prop_planar, RungeKutta() on prop_gnlse, prop_waveguide and prop_free. Multistep() evaluates the nonlinear response once per step; RungeKutta() usually six times, in fewer, longer steps. Multistep() tends to gain where the response dominates a step’s cost, RungeKutta() where it is cheap. Compare the two on a short stretch of your own problem when it matters: the call counts and the time per call are in the run’s step history and log. Choosing a stepper sets out when each is faster, with the measurements.

Many save planes. Landing on every save plane caps the step at the plane spacing. With hundreds of planes, interpolate_planes=True steps across them and interpolates each inside its step instead, and step_planes=k saves the march’s own steps at no extra cost (the solver, “Choosing the saved planes”).

The thread budget#

Every thread pool stays within the process’s thread budget, the smallest of:

its CPU affinity mask, the CPUs it may run on (narrowed by taskset, numactl or a batch scheduler’s allocation);
the physical cores that hold those CPUs, unless SOLPROP_THREADS is set: a core’s second hardware thread shares its arithmetic units, and on 36 cores of two Intel Xeon Gold 6240 one thread per core ran every benchmark case 6 to 31 % faster than two (read on Linux and Windows; Apple silicon has one thread per core);
the tightest Linux control-group CPU quota on its path (containers), rounded down: a quota limits CPU time, and a team of threads that exceeds it is throttled whole, so a quota of 1.5 CPUs runs one thread;
SOLPROP_THREADS, if set.

An explicit count (threads= or a variable below) is clamped to the budget rather than refused; the pools take turns, so each may use all of it. Set SOLPROP_THREADS to each process’s share when several processes share CPUs without being pinned apart. That includes several solprop processes in one CPU-limited container (a Kubernetes CPU limit, Docker --cpus): the quota is the container’s, and each process would otherwise take all of it. A cloud virtual machine or a Slurm job has no quota by default (cat /sys/fs/cgroup/cpu.max prints max); there the CPUs it may run on are the limit. SOLPROP_OVERSUBSCRIBE=1 lets an explicit count exceed the CPUs and the quota (never SOLPROP_THREADS), for a test that compares thread counts on a small machine; a run has no reason to set it. The budget and the limit that set it are recorded in the run’s metadata (thread_budget, thread_budget_source).

These settings apply whatever the pool:

variable	what it sets	default
`SOLPROP_THREADS`	the process’s thread budget, which the native backend’s pools stay within; it may exceed the physical cores, up to the CPUs	the affinity mask, its physical cores and the quota
`SOLPROP_OVERSUBSCRIBE`	`1` lets an explicit count exceed the CPUs and the quota	`0`
`SOLPROP_BLAS_THREADS`	the width of every BLAS other than the native backend’s own while a native backend lives (below)	1
`SOLPROP_FFTW_PLAN`	the native backend’s FFTW planner effort: `estimate`, `measure` or `patient`; any other value is refused	`measure`, except threaded row transforms, which stay at `estimate`
`SOLPROP_FFTW_WISDOM`	a file name, beside which the native backend keeps its FFTW wisdom (below)	none

Free-space node groups. A free-space, line or planar evaluation works through the plane’s columns in node chunks, each paying a fixed cost for its launches, its temporal transforms and its gather and scatter. Their default budget is therefore 1 GiB, against 64 MiB on modal routes; SimulationSettings.node_budget sets it explicitly. On a laptop short of memory a smaller budget saves memory at some speed; on a large device a larger one can still help.

Where a free-space step goes. On the native backend the stages of a free-space evaluation are profiling regions (solprop_native.free.*) that any Kokkos Tools timer reports (profiling).

FFTW planning. measure times candidate algorithms once per transform shape and keeps the fastest; that pays off on long runs, but the timing is only meaningful on an otherwise idle machine, and planning a large multimode problem can take seconds. Use SOLPROP_FFTW_PLAN=estimate for short runs, tests and busy shared machines, and SOLPROP_FFTW_WISDOM=~/.cache/solprop/wisdom to plan once and reuse the plans across runs of the same shapes (the wisdom’s digest is recorded in each run, since plans chosen from different wisdom are different executions). The native backend keeps those plans in a file of its own (below).

The native backend’s pools#

The native backend reads its thread settings once, when solprop_native is first imported:

variable	pool	default
`SOLPROP_NATIVE_THREADS`	the size of Kokkos’ OpenMP pool, an explicit count within the thread budget (above the CPUs only with `SOLPROP_OVERSUBSCRIBE=1`, above `SOLPROP_THREADS` never)	the thread budget
`SOLPROP_NATIVE_KERNEL_THREADS`	response, stage and reduction kernels	sized per operation, up to the cap
`SOLPROP_NATIVE_FFT_THREADS`	FFTW threads for the modal and spatial transforms	sized per plan, up to the cap
`SOLPROP_NATIVE_RAMAN_FFT_THREADS`	threads for the Raman convolution transforms of complete node chunks	sized per plan, up to the cap
`SOLPROP_NATIVE_RAMAN_TAIL_FFT_THREADS`	threads for the Raman transforms of the last, partial node chunk	sized from active rows, up to the cap
`SOLPROP_NATIVE_THREADED_FFT_MIN_SAMPLES`	the real-side samples per execution from which a modal or spatial FFT plan is threaded	16,384 (32,768 on macOS)
`SOLPROP_NATIVE_BAND_TILES`	the field plasma column’s band form: unset for the build’s own (tiled on a GPU, a team per node on a CPU), `0` for the team column, a positive `N` for the tiled column with host teams of `N` threads	unset

The four pool variables are optional positive integers, clamped to the cap; an invalid value raises ValueError at import. An explicit pool setting forces its clamped width and bypasses automatic sizing, including for short FFTs. Setting only the global cap retains automatic sizing. GPU builds ignore these CPU pool settings.

On OpenMP CPU builds, operation dimensions select serial, intermediate or full-width execution. Stage and reduction work, packing and temporal copies, nonlinear responses, native dense transforms, coupled matrix functions and FFT plans have separate size rules. Response sizing accounts for both the total work, active columns and fine samples per column. The choices and plans are cached when the backend is constructed; no runtime tuning occurs during propagation. Partial chunks and fused batch updates use their own work sizes. Vendor BLAS pools retain their process-start settings. Non-OpenMP host builds preserve their execution-space policy; build_info().cpu_size_policy_enabled reports whether operation widths are supported.

The thresholds below are shared across CPU hosts, except that macOS keeps higher stage and FFT starts (8,192 and 32,768), where a short serial loop is still the faster one; C is the clamped capacity of the relevant pool, and sizes count work in the operation itself, not bytes. Between the two thresholds an operation uses min(4, C) threads; equal thresholds give only serial or full-width execution. A kernel operation (any row but the two FFT rows) of a backend that already runs a full-width kernel takes the full pool instead of four threads: GNU OpenMP ends the surplus threads of its team whenever a narrower parallel region follows a wider one and starts them again for the next wide one, which cost a four-thread kernel between full-width ones about half a millisecond each time. FFT plans keep their four threads, since FFTW’s threaded loops run on the runtime’s whole team.

operation	work used for sizing	threading starts	full pool starts
stage, update and reduction	elements in the field or fused batch	4,096 (8,192 on macOS)	as threading
packing and temporal copies	entries actually copied	65,536	65,536
nonlinear response	active columns × components × fine samples	32,768	196,608, or 65,536 fine samples per column, or 40 component columns
native/custom GEMM	output rows × inner dimension × samples	1,048,576	1,048,576
general modal synthesis/projection	active nodes × components × spectral samples × modes	65,536	65,536
coupled matrix functions	active spectral bins × modes³	32,768	32,768
exponential Adams trial	elements in the field × history rows	8,192	65,536
modal/spatial FFT	real-side samples in the actual batch	16,384, 32,768 on macOS (`SOLPROP_NATIVE_THREADED_FFT_MIN_SAMPLES`)	131,072
Raman FFT	real-side samples in the actual batch	65,536	262,144

A response column is one active node and one component; its column-length and column-count rules apply only past its threading start. General modal map updates use the two maps’ combined element count times the polynomial degree. Native GEMM sizing never changes a vendor BLAS’s width or selects a different provider.

The CPU threading measurements explain the thresholds. For example, an 8,192-sample copy remains serial even when a similarly sized arithmetic stage benefits from threading. Short response columns can use four threads while long columns or wider batches use the full pool. The backend applies these choices automatically; start by setting only the process cap when sharing a machine.

python -m solprop_native records the policy constants and explicit override mask. Its existing thread counts remain process capacities. The backend’s metadata records execution_sizing, with the chosen threads and sizing elements for each operation; batch_execution_sizing covers fused batch work. Response entries also report fine_samples and columns, because equal total work can have different column lengths. build_info() reports the resolved process values as requested_threads, runtime_threads, thread_count (kernels), fftw_threads, raman_fft_threads, raman_tail_fft_threads, threaded_fft_min_samples, blas_threads and blas_threading; cpu_size_policy_enabled and explicit_thread_pools (a bit mask: kernel 1, FFT 2, Raman 4, Raman tail 8) distinguish automatic sizing from explicit requests.

Free-space transforms are planned at SOLPROP_FFTW_PLAN’s effort (measure by default); estimate skips the measured planning for short runs and tests. Answers at different thread counts may differ by rounding; solprop holds them to its rounding bounds, not to identical bits (SEP 25). Contiguous modal and Raman FFTW plans default to measured planning at one thread and estimate planning at wider widths; the explicit effort overrides both, including long Raman rows and partial chunks. Measured planning can add substantial construction time even when the warmed propagation is faster. When construction dominates a short run, keep the plans in a wisdom file (below) rather than give up measuring them; SOLPROP_FFTW_PLAN=estimate skips planning altogether, but its plans ran one-thread windows up to 1.6 times slower on an AMD EPYC 7763. On macOS with NEON FFTW, serial and long Raman rows use Kokkos-FFT, while parallel power-of-two rows of at most 16,384 samples retain vDSP. The build guide describes the coherent dependency recipe and the fallback for a library without NEON codelets.

Reusing the native backend’s plans. SOLPROP_FFTW_WISDOM keeps the native backend’s measured plans: a backend imports them before it plans and adds the plans it measured after. Nothing is read or written unless the variable names a file. The native backend links its own FFTW, so its plans go to a file of their own beside the named one, not to the named file itself: <name>.native-<key>, whose key digests the FFTW library’s version and configuration and the processor model: machines of different kinds that share a home directory keep separate plans, and FFTW refuses wisdom another build of it wrote. The processes of a scan can share one file. Readers take no lock, since the file is only ever replaced whole; a writer takes <file>.lock, merges what others stored, and writes only when that adds a plan. A file that cannot be read or written is reported in the log and costs only the measurement. Wisdom chooses among FFTW’s exact algorithms, so it changes no result beyond the rounding a fresh measurement can change; the run’s metadata records wisdom_loaded and the wisdom_digest of the plans in force. For a scan of one-thread jobs it removes most of each job’s construction: on a Xeon Gold 6240, constructing the averaged helium capillary took 1.40 s cold and 0.03 s with the wisdom in place, beside a 0.27 s window (1.78 s and 0.05 s on an AMD EPYC 7763), and the comb-to-continuum capillary 5.0 s and 0.06 s (measurements). For example:

export SOLPROP_FFTW_WISDOM=~/.cache/solprop/wisdom

The first job of each shape measures and stores its plans; every later one, in any process, finds them. Threaded row transforms are planned at estimate unless SOLPROP_FFTW_PLAN asks otherwise, so they have no measured plans to keep; but plans that a run under an explicit measure stored for a width are found by later runs at that width, whatever their effort.

OMP_NUM_THREADS and KOKKOS_NUM_THREADS do not size the Kokkos pool — the backend initialises Kokkos with an explicit thread count, which takes precedence; set SOLPROP_NATIVE_THREADS instead. (Kokkos prints a warning at start-up if OMP_NUM_THREADS asks for fewer threads than it was given.)

The default cap is the thread budget (above): the CPUs in the process’s affinity mask, a container’s quota and SOLPROP_THREADS. Automatic sizing avoids assigning all of those threads to small operations. On a shared machine or when several solprop processes run side by side, set SOLPROP_THREADS or SOLPROP_NATIVE_THREADS to each process’s CPU budget. The operation policy stays within that budget.

OpenMP placement and waiting#

The native CPU backend runs on OpenMP, whose standard variables apply:

OMP_PROC_BIND and OMP_PLACES pin threads to cores, which avoids threads migrating between cores and sockets: OMP_PROC_BIND=spread OMP_PLACES=cores is a good default for one run on a multi-socket machine. When none of OMP_PROC_BIND, OMP_PLACES, GOMP_CPU_AFFINITY and KMP_AFFINITY is set, the native backend’s import sets OMP_PROC_BIND=false: the threads stay unpinned, as the runtimes leave them anyway, and Kokkos does not print its notice that the variable is unset, whose advice is for a run with the machine to itself. The run’s metadata records what applied as openmp_proc_bind. With a build whose FFTW uses pthreads rather than OpenMP, prefer OMP_PROC_BIND=false. Binding also holds back SciPy’s transforms on the host, which a free-space run’s launch and edge reads use: on 16 cores of an AMD EPYC 7763, a transform of 384² × 256 took 0.48 s with 16 workers under OMP_PROC_BIND=close OMP_PLACES=cores, as long as with one worker, and 0.11 s unbound.
OMP_WAIT_POLICY decides what idle threads do between parallel regions: active keeps them spinning, passive lets them sleep at once. The native backend chooses for you when none of OMP_WAIT_POLICY, GOMP_SPINCOUNT and KMP_BLOCKTIME is set. It runs several thread teams in turn (Kokkos’, the threaded FFTs’, the BLAS’s), and when its threads fill the CPUs one team’s spinning threads take the cores the next needs: on 16 cores of an AMD EPYC 7763 the runtime’s own waiting made the four-mode capillary 25 times slower (334 s against 13.5 s). So the import sets OMP_WAIT_POLICY=passive. The setting stays in the environment, so worker processes wait alike, and the run’s metadata records what applied as openmp_wait_policy. Where hardware threads are spare and nothing else will use them, spinning is faster: on 36 cores of two Intel Xeon Gold 6240 (72 hardware threads), with the whole machine to itself, OMP_WAIT_POLICY=active ran the supercontinuum in 49 s against 94 s and the capillary in 9.7 s against 14.7 s. It is not the default because the runtime reads the policy once, at start-up, and work that arrives on the machine later takes those spare threads; set it for a run on a machine it has to itself. LLVM’s OpenMP runtime (used on macOS) also honours KMP_BLOCKTIME, the time a thread spins before sleeping; do not set it for GNU OpenMP.
On cores your run has to itself, such as a batch job’s allocation, also pinning the threads helps: with idle threads sleeping, OMP_PROC_BIND=close OMP_PLACES=cores ran the tracking cases 11 % faster (geometric mean) on 16 cores of a shared AMD EPYC 7763 node, the comb-to-continuum and graded-index cases 28 % and the radial filament 17 %, while on a dedicated socket of a Xeon Gold 6240 it changed little (1 % slower overall). Keep close: spread scatters a four-thread kernel over the EPYC’s core complexes. Leave it unset where several processes share a machine, since each would pin its threads to the same first cores.

BLAS and vendor libraries#

The BLAS matters for the dense modal transforms of multimode capillaries, coherent walls and the general modal route. A BLAS with its own thread pool beside the native backend’s oversubscribes the cores: with a pthreads OpenBLAS at the full core count, the modal and wall cases ran 10 to 75 times slower than necessary. The Linux distribution build of the native backend therefore carries an OpenBLAS built with OpenMP, which shares the backend’s runtime. At start-up the backend fits a host OpenBLAS to its own pool unless OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS or OMP_NUM_THREADS is set: a pthreads build is held to one thread, an OpenMP build capped at the Kokkos thread count. NumPy’s own BLAS, a separate library beside the native backend’s, serves only small host-side products during a native run, yet its idle workers spin on the cores the native pool needs: while a native backend lives, every BLAS threadpoolctl finds other than the backend’s own is held to one thread (SOLPROP_BLAS_THREADS sets another count), and the last backend to close gives the pools back. The run’s metadata records what applied as host_blas.

Guide construction and coupling happen before a backend exists. They also hold the controllable host BLAS pools at one thread, or at SOLPROP_BLAS_THREADS within the process budget. The hold covers ModalGuide.from_provider, a standalone graded-index mode solve (including a node rule’s tail certificate), general_modal_route, kernel assembly, FieldPlane.from_pulse and couple. The general modal route builds its large reference tables inside Simulation.from_waveguide, before kernel assembly; an incident launch builds its plane’s radial transform before coupling. This overrides OPENBLAS_NUM_THREADS and the other vendor defaults for those calls, then restores the previous limits even if construction fails. A native backend’s own BLAS keeps its fitted width. Nested or overlapping construction shares the first scope’s hold. Construction and live backend holds share each library’s ownership: the narrowest active request applies, and the last owner restores the original width, even if the calls start and finish in different orders. Separate processes, including scan workers, keep their own pools. Accelerate exposes no runtime thread control to threadpoolctl, so these scopes leave it unchanged.

Outside a backend’s lifetime and these construction scopes, a BLAS’s own variables apply:

library	thread variable	notes
OpenBLAS (pthreads build)	`OPENBLAS_NUM_THREADS`	the usual Linux default
OpenBLAS (OpenMP build)	`OMP_NUM_THREADS`	ignores `OPENBLAS_NUM_THREADS`; the best choice beside the native backend, which shares its runtime
Intel MKL	`MKL_NUM_THREADS`, or per domain `MKL_DOMAIN_NUM_THREADS`	e.g. `MKL_DOMAIN_NUM_THREADS="MKL_DOMAIN_BLAS=1,MKL_DOMAIN_FFT=8"`
Apple Accelerate	`VECLIB_MAXIMUM_THREADS`	threads through Grand Central Dispatch
AMD AOCL-BLIS	`BLIS_NUM_THREADS`

NumPy’s own BLAS (used by numpy_ref and by solprop’s translation layer) answers to the same variables. A locally built native backend links a vendor BLAS (Accelerate on macOS, MKL when oneAPI is available on Linux, otherwise OpenBLAS); its build record says which (python -m solprop_native).

One OpenMP runtime per process. The native backend, FFTW and some BLAS builds each use OpenMP, and two different OpenMP runtimes loaded into one process can crash or oversubscribe the machine. solprop does not use pyFFTW, whose wheels bring a runtime of their own, but a script or another package may. On macOS solprop_native must then be imported first, and raises an error otherwise: put import solprop_native at the top of such a script (OpenMP runtimes).

Warning

On Windows on Arm, setting OMP_NUM_THREADS, OPENBLAS_NUM_THREADS or GOTO_NUM_THREADS to a small number can crash or hang NumPy’s own linear solves, with solprop not involved. On NumPy 2.5.2, widths 1 and 2 failed at matrix sizes 23 to 31, and width 3 at size 31. Leave them unset. solprop warns on import if it finds such a value. threadpoolctl limits were safe at the same sizes, so Pool workers on this platform use those instead of the variables.

Recipes#

One large run on a workstation. Give the run every performance core, pin it, and let idle threads sleep if you are using the machine for anything else:

SOLPROP_NATIVE_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores OMP_WAIT_POLICY=passive \
    python my_run.py

numpy_ref runs on one thread apart from its matrix products, and no setting changes that.

How far a single run scales. A single-mode fibre on a few thousand time samples gains little beyond two to four threads: its transforms are short and each step is mostly synchronisation. Multimode capillaries and the general modal route, whose products and node loops are large, keep gaining to eight to eighteen threads. Free-space runs with large transverse grids scale furthest.

A scan of many runs. For a scan over pressure, energy or noise seeds, many single-threaded workers almost always beat one many-threaded run: each worker keeps its own caches and nothing waits for synchronisation. solprop’s Pool executor does this for you: every worker is a newly spawned Python process started with all of SOLPROP_THREADS, OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS and VECLIB_MAXIMUM_THREADS set to its threads (one by default) and OMP_WAIT_POLICY=passive, so every pool is sized within that budget, and on Linux each worker is pinned to its own CPUs. The number of workers is the number of CPUs available divided by threads, reduced if a memory probe of the first point shows that fewer fit (four fifths of the memory, less the controller’s own, divided by one worker’s footprint). run_scan runs its points one after another in the calling process unless it is given an executor, so ask for the pool, in the script or on the command line:

python my_scan.py                            # the script calls run_scan(..., executor=Pool())
python -m solprop.scan my_scan.py --workers 8    # or let the launcher choose Pool

A problem too large to fit many copies in memory can use fewer, wider workers, Pool(threads=4) (or --threads 4). To run two scans side by side on one machine, give each its own CPUs with SOLPROP_SCAN_CPU_OFFSET and SOLPROP_SCAN_CPU_COUNT. See scans and batches. Stay on the native backend at one thread per worker too, and time one point of the scan with your real number of workers before a large campaign.

If you manage the parallelism yourself — a job array, GNU parallel, several scripts started by hand — each process needs every pool held to one thread explicitly, before Python starts:

SOLPROP_THREADS=1 OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
VECLIB_MAXIMUM_THREADS=1 OMP_WAIT_POLICY=passive OMP_PROC_BIND=false \
    python point.py

On Windows on Arm, leave out OMP_NUM_THREADS and OPENBLAS_NUM_THREADS, for the reason under BLAS and vendor libraries.

On a multi-socket server, keep a multi-threaded run’s threads within one socket (OMP_PLACES=cores with OMP_PROC_BIND=close, or taskset/numactl): crossing sockets usually costs more in memory traffic than the extra cores return for the problem sizes solprop runs, and two runs, one per socket, are usually faster than one run across both.

A cluster. Slurm writes an array job whose tasks each run a Pool inside their allocation; it respects SLURM_CPUS_PER_TASK, and its threads sets the same variables in every task.

A GPU. On a CUDA or HIP build of the native backend the CPU pools do not matter; the backend uses one device per process, chosen with CUDA_VISIBLE_DEVICES or HIP_VISIBLE_DEVICES. Small problems are limited by kernel-launch latency and often run faster on the CPU: run them with the CPU wheel in a second environment. Run independent problems in separate processes to use several devices. A job set up for a device names the requirement, and stops before it starts if the environment has lost its GPU build (a job that must run on a device):

CUDA_VISIBLE_DEVICES=0 SOLPROP_REQUIRE_DEVICE=cuda python my_run.py

See The native backend.

A processor without AVX2. An x86-64 wheel loads its x86-64-v2 build there, which takes about 20 to 25 % longer by geometric mean; solprop_native.build_info() reports which build is loaded, and SOLPROP_NATIVE_CPU_VARIANT=x86-64-v2 asks for that build on a faster processor, for comparisons (The native backend).

Memory. Large multimode and general modal problems can be limited by memory before threads; how to size and shrink them is described in Execution resources.

Checking what was used#

The backend used is logged at INFO level, and the pools it chose are in the run’s backend record, which is saved with the output:

import solprop
from solprop.driver import prop_capillary

solprop.configure_logging("INFO")  # prints "Using native backend", among others
run = prop_capillary(
    125e-6,
    1e-3,
    "He",
    1.0,
    wavelength=800e-9,
    wavelength_limits=(200e-9, 2e-6),
    trange=100e-15,
    fwhm=10e-15,
    energy=1e-9,
    n_save=2,
    progress="none",
    backend="native",
)
settings = run.metadata.backend.settings
print(
    {key: settings[key] for key in ("runtime_threads", "thread_count", "fftw_threads")}
)

The same record also carries raman_fft_threads, blas_library and the build profile; python -m solprop_native prints the whole build record without running anything.

Profiling the native backend#

Kokkos profiling hooks are compiled into every native build, and every kernel, region and allocation carries a solprop_native.* label (for example solprop_native.transform.*, solprop_native.raman.*, solprop_native.plasma.*, solprop_native.statistics.*). Load any Kokkos Tools plugin at run time with KOKKOS_TOOLS_LIBS; nothing is linked into the package. For the simple kernel timer:

git clone --depth 1 --branch 5.2.0 https://github.com/kokkos/kokkos-tools.git
cmake -S kokkos-tools -B kokkos-tools/build -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$PWD/kokkos-tools/install"
cmake --build kokkos-tools/build --target install --parallel

KOKKOS_TOOLS_LIBS="$PWD/kokkos-tools/install/lib/libkp_kernel_timer.so" \
  python my_simulation.py

On macOS the plugin is libkp_kernel_timer.dylib. At shutdown the timer reports call counts and accumulated times for the solprop_native.* labels and for Kokkos-FFT’s KokkosFFT::exec_plan regions; the NVTX and ROCTX connectors pass the same labels to Nsight Systems and the ROCm profilers. The free-space source evaluation’s stages are regions of their own (solprop_native.free.core, .transverse_inverse, .pack, .to_time, .clear_projected, .response, .to_frequency, .transverse_forward); with no tool loaded a region costs one flag test. For the Runge–Kutta march, the backend’s metadata()["source_evaluations"] counts the native source evaluations since the initial state was set, rejected trials included.