solprop.output.hdf5#

The streaming HDF5 output handler, with cache and resume.

One file per run, kept open between optional snapshot transactions. Saves and metadata calls reuse the handle. Without snapshots the file opens only once.

What is on disk, and where. Root attributes carry the identifying scalars – schema_version, spectral_convention, field_layout, configuration_hash, simulation_kind, grid_kind, created and, once a backend has executed the run, backend – and meta/json carries the whole solprop.output.schema.RunMetadata as one JSON string. The metadata goes in a dataset rather than an attribute because HDF5 caps an attribute in the object header at 64 kB and a serialised configuration – a capillary with a tabulated cladding glass, a stored noise realisation – goes past that; the identifying scalars stay attributes, where a reader looks for them first. field is (n_z, n_modes, n_pol, n_omega) complex, chunked one plane at a time and extendable along n_z, in the optics convention and marked so. noise_phase, when the run has noise, has the same shape and stays in the internal convention. z is the plane positions, stats/<name> one dataset per statistic, each with a leading plane axis, and step_history/<column> the attempts of the whole run concatenated, with step_history/counts saying how many belong to each plane.

Preallocation and the saved-plane count. The plane-indexed datasets are created at the first save for every plane the run reserved (reserve()), with an unlimited leading axis, and grow by doubling if the run saves more; the step-history and between-planes columns grow by doubling too. The root attribute n_saved counts the planes actually written: each plane’s data is written and flushed first and the count last, so a hard stop leaves the count at the last complete plane. The attempts and rows belonging to those planes are the sums of step_history/counts and step_stats/counts over them, so the count alone fixes every column’s length. Every reader and resume() read only the first n_saved planes. A clean close trims every dataset to its count, so a finished file holds no unused rows.

Field retention (SEP 22). The root attribute fields says which planes keep their field (FieldRetention), and field_shape the shape of one plane’s field whether or not it is stored. A run that keeps every field stores field and noise_phase as above. One that keeps the named planes stores exactly those, in plane order, with a field_planes dataset of their indices beside them; one that keeps none has no field and no noise_phase dataset. resume() continues only from a plane that carries a field: a terminal checkpoint always does, and a file whose last regular plane is field-free and which holds no checkpoint is refused by name when a run is continued from it.

Cache and resume. check_cache() compares the file’s configuration hash with the requested one (ADR-066): they must be equal, so resuming with different physics or numerics of the same shape is refused, by naming what changed. A file of another output schema is not read at all (SCHEMA_VERSION). resume() returns the metadata, the last plane – converted back to the internal convention – and the step history. A handler built with append=True then carries on in the same file: it opens it read-write instead of replacing it, compares the record offered with the file’s, holds the file’s record as its own, and continues numbering planes and attempts where the file left off. open_file() opens a finished file read-only through the same protocol it was written with, which is how solprop.processing reads a run after the driver has closed it.

Compression is restricted to what h5py ships: None, 'gzip' or 'lzf'. Blosc is not offered, because it needs a filter plugin solprop does not depend on.

The manual page is docs/guide/results/output.md.

Module Attributes

COMPRESSIONS

Compression filters h5py ships with; no plugin, so no Blosc (dependencies.md).

Functions

`check_cache`(path, metadata)	Say whether a file may be continued as the run `metadata` describes.
`open_file`(path)	Open a finished run for reading, through the protocol it was written with.
`read_recipe`(file)	Return the recipe an open file records, or `None` for a run from a script.
`resume`(path)	Read a file's resumable state and cumulative work, preferring its checkpoint.
`write_recipe`(file, record)	Write the recipe group of an open file.

Classes

`HDF5Output`(path, *[, compression, ...])	Stream a run to one HDF5 file, opened once and extended plane by plane.
`ResumeState`(metadata, plane, n_saved, history)	Everything a driver needs to carry on from a file it wrote earlier.

solprop.output.hdf5.COMPRESSIONS = (None, 'gzip', 'lzf')#: Compression filters h5py ships with; no plugin, so no Blosc (dependencies.md).

class solprop.output.hdf5.HDF5Output(path, *, compression=None, compression_opts=None, append=False, readonly=False)[source]#

Bases: PlaneStore

Stream a run to one HDF5 file, opened once and extended plane by plane.

Reading through __getitem__ and fields() works while the handler is open and stops when it is closed, because the arrays live in the file rather than in memory; open_file() opens a finished file for reading afterwards.

Parameters:

pathpathlib.Path or str: Where to write. An existing file is replaced, with a warning, when the run opens.
compression{None, ‘gzip’, ‘lzf’}, optional: Dataset compression. 'lzf' is fast and modest; 'gzip' is slower and smaller. Nothing else: a Blosc filter would need a plugin that solprop does not depend on.
compression_optsint or None, optional: Level for 'gzip', 0 to 9; not accepted for the other choices.
appendbool, optional: Continue the run the file already holds instead of replacing the file: the resume path, after resume() has handed the driver the last plane. The record given to open is refused when its configuration hash differs from the file’s; the file’s own record is kept, creation time included, and becomes the handler’s metadata, and the run’s layout is read from the file so that the next plane is checked against the ones already there.
readonlybool, optional: Open the file at once, for reading only: the handler comes back open, with the file’s own record as its metadata, and refuses open and save. This is what open_file() builds.

Raises:

OutputError: If the compression request is not one h5py can serve, if both append and readonly are asked for, or if a read-only file cannot be read.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import HDF5Output
>>> from solprop.output.schema import (
...     GridRecord, RunMetadata, SavedPlane, StepperRecord
... )
>>> from solprop.stepper import DOPRI5, PIController
>>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12)
>>> metadata = RunMetadata(
...     grid=GridRecord.from_grid(grid),
...     modes=(),
...     simulation_kind="mode_averaged",
...     configuration={},
...     stepper=StepperRecord.from_settings(
...         DOPRI5, PIController(rtol=1e-8, atol=0.0)
...     ),
... )
>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5", compression="lzf")
...     handler.open(metadata)
...     start = np.ones((1, 1, grid.n_omega), complex)
...     handler.save(SavedPlane(z=0.0, field=start))
...     shape = handler["field"].shape
...     handler.close()
>>> shape
(1, 1, 1, 2560)

close()[source]#

Close the owned handle, including a failed post-run annotation handle.

Repeated calls are harmless. A failed close retains the handle and writer ownership for a later retry, even if propagation already closed the store.

Raises:

OutputError: If the handle cannot be closed safely.

open(metadata, *, force=False)[source]#

Begin a run, recording what produced it; refused on a read-only handler.

Parameters:

metadataRunMetadata: The record of the run about to start.
forcebool, optional: Whether to overwrite metadata already held for a different run.

Raises:

OutputError: If the handler was built with readonly=True.

property path: Path[source]#

Return where the file is written.

Returns:

pathlib.Path: The path given to the constructor.

reopen()[source]#

Rebind a suspended writer without beginning a new execution.

Raises:

OutputError: If the file changed while suspended or cannot be reopened safely.

save(plane)[source]#

Store one saved plane and flush it to disk; refused on a read-only handler.

The flush is what makes a hard stop – a SIGKILL, an out-of-memory kill, a lost node – lose at most the plane being written: the plane’s data is flushed before the file’s n_saved count is raised, so a plane cut off half-way is not counted, and readers do not see it. An ordinary interruption (KeyboardInterrupt, an error) never needed it: the run loop closes the handler in a finally and closing writes everything. One flush per plane costs nothing against the steps between planes.

Parameters:

planeSavedPlane: The position, field, statistics and step history to store.

Raises:

OutputError: If the handler was built with readonly=True.

save_march(snapshot)[source]#

Keep the Adams march’s state at the position just saved, for a resume.

One snapshot is kept, rewritten in place at every call, beside the planes and outside the record: it says where the march stood, not what was computed, and a file without one resumes by starting the history afresh.

Parameters:

snapshotsolprop.output.schema.MarchSnapshot: The march’s state at the last saved plane or terminal checkpoint.

Raises:

OutputError: If the handler was built with readonly=True or is not open.

suspend()[source]#

Close the HDF5 handle while retaining the live writer and its ownership.

Raises:

OutputError: If the writer is not open or its handle cannot be closed safely.

class solprop.output.hdf5.ResumeState(metadata, plane, n_saved, history, outcome=None, checkpoint=None, march=None)[source]#

Bases: Displayed

Everything a driver needs to carry on from a file it wrote earlier.

Equality is by identity (eq=False) because the plane holds arrays.

Attributes:

metadatasolprop.output.schema.RunMetadata: The record of the interrupted run, including its step schedule.
planesolprop.output.schema.SavedPlane: The accepted terminal checkpoint when present, otherwise the last regular plane, with its field back in the internal FFT convention. Modal amplitudes remain in output units until the driver applies the weight at z.
n_savedint: Number of regular planes stored. The next scheduled index is determined from the actual position, independently of this count.
historysolprop.output.schema.StepHistory: Every attempt of the whole run so far, concatenated.
outcomesolprop.output.schema.RunOutcome or None: Completion status and cumulative call ledgers; None when the file records none.
checkpointsolprop.output.schema.TerminalCheckpoint or None: Separate accepted terminal state, preferred over the last regular save.
marchsolprop.output.schema.MarchSnapshot or None: The Adams march’s state where the run stopped, when the file kept one there.

Examples

>>> from solprop.output.hdf5 import ResumeState
>>> ResumeState.__doc__.splitlines()[0]
'Everything a driver needs to carry on from a file it wrote earlier.'

checkpoint: TerminalCheckpoint | None = None#

history: StepHistory#

property interpolated: bool[source]#

Whether the plane the file resumes from was interpolated inside a step.

A landed plane is the end of an accepted step of the run; an interpolated one lies inside a step, and a resume from it needs resume="interpolated". The entrance is the initial field.

Returns:

bool: Whether the plane is not a state the march passed through.

march: MarchSnapshot | None = None#

metadata: RunMetadata#

n_saved: int#

outcome: RunOutcome | None = None#

plane: SavedPlane#

solprop.output.hdf5.check_cache(path, metadata)[source]#

Say whether a file may be continued as the run metadata describes.

The resume gate (ADR-066): the two records’ configuration hashes must be equal. A refusal names the fields that differ. The checks that depend on the file’s state, such as whether any work is left, are the driver’s.

Parameters:

pathpathlib.Path or str: The file to check.
metadataRunMetadata: The record of the run about to start.

Returns:

bool: True when the file may be continued; False when there is no file to resume from.

Raises:

OutputError: If the hashes differ, or the file was written at another output schema; or if the file’s general modal record names an accessor this build no longer admits (_check_accessors).

Examples

>>> from solprop.output.hdf5 import check_cache
>>> from solprop.output.schema import _example_metadata
>>> check_cache("no-such-file.h5", _example_metadata())
False

solprop.output.hdf5.open_file(path)[source]#

Open a finished run for reading, through the protocol it was written with.

The handler comes back open and read-only: __getitem__, fields, stat_names, n_saved and metadata answer from the file, open and save are refused, and close releases the file – a with block does that on the way out. It is how solprop.processing reads a file after the run that wrote it has ended; resume() reads the last plane only, and a handler built with append=True opens the file read-write to continue the run.

Parameters:

pathpathlib.Path or str: The file to read.

Returns:

HDF5Output: An open, read-only handler.

Raises:

OutputError: If the file cannot be opened, or carries no metadata.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import MemoryOutput, open_file
>>> from solprop.output.schema import (
...     GridRecord, RunMetadata, SavedPlane, StepperRecord
... )
>>> from solprop.stepper import DOPRI5, PIController
>>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12)
>>> memory = MemoryOutput()
>>> memory.open(RunMetadata(
...     grid=GridRecord.from_grid(grid), modes=(), simulation_kind="mode_averaged",
...     configuration={},
...     stepper=StepperRecord.from_settings(
...         DOPRI5, PIController(rtol=1e-8, atol=0.0)
...     ),
... ))
>>> memory.save(SavedPlane(z=0.0, field=np.ones((1, 1, grid.n_omega), complex)))
>>> with tempfile.TemporaryDirectory() as folder:
...     memory.to_file(Path(folder) / "run.h5")
...     with open_file(Path(folder) / "run.h5") as stored:
...         shape, planes = stored["field"].shape, stored.n_saved
>>> shape, planes, stored.closed
((1, 1, 1, 2560), 1, True)

solprop.output.hdf5.read_recipe(file)[source]#

Return the recipe an open file records, or None for a run from a script.

The recipe group of an optical file or a scan file (SEP 41, S2).

Parameters:

fileh5py.File: The open file.

Returns:

RecipeRecord or None: The record.

solprop.output.hdf5.resume(path)[source]#

Read a file’s resumable state and cumulative work, preferring its checkpoint.

The last plane comes back in the internal FFT convention, through solprop.fourier.spectrum_to_internal(), so a driver can hand it straight to a backend without conjugating anything itself. A separate terminal checkpoint contributes no regular observation but supplies its actual position, pending attempt history and cumulative call ledgers. Reading preserves its landing classification; the driver requires explicit resume="interpolated" consent before restarting from an interpolated field.

A run that keeps no field on its last regular plane (SEP 22) comes back with that plane’s field=None unless a terminal checkpoint, which always carries the field, follows it; the driver refuses to continue from it by name.

Parameters:

pathpathlib.Path or str: The file to read.

Returns:

ResumeState: The metadata, the last plane, how many planes the file holds, and every attempt the stepper made.

Raises:

OutputError: If the file cannot be read, carries no metadata, or holds no plane.

Examples

>>> import numpy as np, tempfile
>>> from pathlib import Path
>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output import HDF5Output
>>> from solprop.output.hdf5 import resume
>>> from solprop.output.schema import SavedPlane, _example_metadata
>>> metadata = _example_metadata()
>>> n_omega = metadata.grid.n_omega
>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5")
...     handler.open(metadata)
...     handler.save(SavedPlane(z=0.25, field=np.ones((1, 1, n_omega), complex)))
...     handler.close()
...     state = resume(Path(folder) / "run.h5")
>>> state.n_saved, state.plane.z
(1, 0.25)

Carry on in the same file with a handler built with append=True:

>>> with tempfile.TemporaryDirectory() as folder:
...     handler = HDF5Output(Path(folder) / "run.h5")
...     handler.open(metadata)
...     handler.save(SavedPlane(z=0.0, field=np.ones((1, 1, n_omega), complex)))
...     handler.close()
...     again = HDF5Output(Path(folder) / "run.h5", append=True)
...     again.open(metadata)
...     again.save(SavedPlane(z=0.5, field=np.ones((1, 1, n_omega), complex)))
...     again.close()
...     resume(Path(folder) / "run.h5").n_saved
2

solprop.output.hdf5.write_recipe(file, record)[source]#

Write the recipe group of an open file.

Parameters:

fileh5py.File: The file, open for writing, without a recipe group.
recordRecipeRecord: The recipe.