Profiling Struphy with scope-profiler

This tutorial shows two ways to use scope-profiler:

1. Profile a small standalone Python workload with ProfileManager region timing.

2. Configure a Struphy Simulation with ProfilingOptions, including line profiling, and activate profiling at sim.run(...).

The same HDF5 output file can be inspected from Python, from the command-line interface, or in the terminal TUI.

Imports and output directory

Use a small temporary directory so repeated notebook runs do not overwrite production simulation data.

import shutil
import tempfile
from pathlib import Path

import numpy as np
from scope_profiler import ProfileManager

workdir = Path(tempfile.mkdtemp(prefix="struphy_profile_tutorial_"))

1. Profile standalone Python code

ProfileManager.profile_region(...) profiles a with block. @ProfileManager.profile(...) profiles a decorated function. This first example only records region timings, which is enough for coarse timing and produces a small HDF5 profile file.

@ProfileManager.profile("demo: matrix multiply")
def matrix_work(n: int) -> float:
    a = np.arange(n * n, dtype=float).reshape(n, n)
    b = a.T.copy()
    c = a @ b
    return float(c.sum())


profile_file = workdir / "demo_scope_profile.h5"

with ProfileManager.session(
    file_path=str(profile_file),
    return_results=True,
    verbose=False,
) as prof:
    with ProfileManager.profile_region("demo: total"):
        total = 0.0
        for n in (16, 24, 32):
            total += matrix_work(n)
        print(f"Total sum: {total:.2f}")

results = prof.results

The context manager finalizes the run and writes one HDF5 file. With return_results=True, the finalized result object is also available in memory.

results.print_summary(title="Standalone demo", include=[r"^demo:"])

The same file can be inspected from Python. The CLI and TUI commands are shown later for terminal workflows.

from scope_profiler import read_h5

standalone_results = read_h5(profile_file)
standalone_results.print_summary(title="Standalone HDF5 summary", include=[r"^demo:"])

2. Profile a Struphy simulation

Struphy instruments the important phases of Simulation.run() with ProfileManager: allocation and setup, the time-stepping loop (model.integrate), propagators and solvers, diagnostics, particle sorting, and output. A profiling session is only active when sim.run(profiling_activated=True) is called. This means the same Simulation setup can be used for an ordinary run or a measured run.

For a useful performance experiment, keep the problem small enough to run quickly but make the time-stepping loop execute several times. Include a warm-up run when comparing configurations: compilation, cache creation, allocation, and filesystem setup can otherwise dominate the measurement. Use save_step to avoid measuring a workload dominated by output. Start with region timing; enable use_line_profiler=True only after the expensive region is known, because line profiling adds substantial overhead.

from struphy import EnvironmentOptions, ProfilingOptions, Simulation, Time, grids
from struphy.models import Poisson

env = EnvironmentOptions(
    out_folders=str(workdir),
    sim_folder="poisson_profile_demo",
    # Saving every step is useful for a demo, but can dominate a benchmark.
    save_step=1,
)

profiling_opts = ProfilingOptions(
    file_path=str(workdir / "poisson_profile.h5"),
    # First locate the expensive region with region timing.
    use_line_profiler=False,
)

sim = Simulation(
    model=Poisson(),
    env=env,
    time_opts=Time(dt=0.01, Tend=0.01),
    grid=grids.TensorProductGrid(num_elements=(4, 4, 1)),
    profiling_opts=profiling_opts,
)

# This profiles the complete Simulation.run() call, including setup,
# model.integrate(), diagnostics, and data output.
sim.run(profiling_activated=True)
struphy_profile_file = Path(profiling_opts.file_path)

from scope_profiler import read_h5

profile_results = read_h5(struphy_profile_file)
region_names = sorted(region.name for region in profile_results.get_regions())
print(", ".join(region_names))

Measure setup and time stepping separately

The profile contains both one-time setup and repeated work. For a performance experiment, inspect them separately: setup is useful for allocation and initialization studies, while model.integrate and its children describe the cost per time step. For a longer benchmark, increase Tend so several steps are recorded, and set save_step larger when output is not part of the question. Include a warm-up run before comparing configurations because compilation, cache creation, allocation, and filesystem setup can otherwise dominate the measurement.

capture_full_output(
    profile_results.print_summary,
    title="What did Simulation.run() measure?",
    include=[r"^setup:", r"^model\.integrate$", r"^prop: ", r"^solve:", r"^kernel: ", r"^save data$"],
)

Line-profile output for selected Struphy regions

To get line-by-line timing, repeat the run with use_line_profiler=True (and preferably a shorter Tend). The HDF5 file then contains line-profile records for decorated functions and profiled regions. The filter below prints a representative solver region:

• setup: total, the top-level setup block in Simulation.run.

• solve: PoissonSolve, the Poisson propagator solve call.

The source column is shown because these regions come from real Struphy source files.

line_profile_opts = ProfilingOptions(
    file_path=str(workdir / "poisson_line_profile.h5"),
    use_line_profiler=True,
)
line_sim = Simulation(
    model=Poisson(),
    env=EnvironmentOptions(out_folders=str(workdir), sim_folder="poisson_line_demo", save_step=1),
    time_opts=Time(dt=0.01, Tend=0.01),
    grid=grids.TensorProductGrid(num_elements=(4, 4, 1)),
    profiling_opts=line_profile_opts,
)
line_sim.run(profiling_activated=True)
line_profile_file = Path(line_profile_opts.file_path)

from scope_profiler.line_profile_cli import print_line_profile

print_line_profile(
    line_profile_file,
    region=r"^prop: PoissonSolve",
    display_html=True,
)

For a parameter file generated by struphy params <ModelName>, the same pattern is:

profiling_opts = ProfilingOptions(
    file_path="my_profile.h5",
    use_line_profiler=True,
)

sim = Simulation(..., profiling_opts=profiling_opts)
sim.run(profiling_activated=True)

Omit the profiling_activated keyword for normal production runs without profiling overhead.

3. Plot and inspect the profile

There are three useful levels of inspection: the summary table finds expensive regions; the duration chart ranks them; and the Gantt/flame views explain when they run and how parent/child time is composed. A region’s inclusive duration includes its children, so do not add the model.integrate, propagator, solver, and kernel bars together unless you intentionally want overlapping costs.

scope-profiler also provides a CLI and a terminal TUI for interactive work outside the notebook:

scope-profiler inspect profiling_data.h5
scope-profiler line-profile profiling_data.h5
scope-profiler plot durations profiling_data.h5 -o figures --sort-by total --top-n 20
scope-profiler plot durations profiling_data.h5 -o figures --stack-children --sort-by total --top-n 12
scope-profiler plot gantt profiling_data.h5 -o figures --min-duration 0.0001 --collapse-depth 4
scope-profiler plot flame_graph profiling_data.h5 -o figures
scope-profiler plot flame_chart profiling_data.h5 -o figures
scope-profiler plot quick profiling_data.h5 -o figures
scope-profiler tui profiling_data.h5

We recommend checking out the scope-profiler documentation for details on postprocessing using the tool.

The --stack-children option splits each duration bar into self time plus direct children. Use --collapse-depth and --min-duration to make a busy Gantt chart readable. flame_graph aggregates call stacks; flame_chart preserves their time position.

IPython magics

The magics are useful for quick notebook experiments and comparisons. %%scope records a notebook cell as one named region; %scope_load imports the HDF5 profile produced by sim.run() into the same in-memory registry. Use the Struphy HDF5 profile for detailed nested run data, and magics for small local measurements or before/after comparisons.

%load_ext scope_profiler.ipython_magics

%%scope notebook_overhead -p
sum(i * i for i in range(10_000))

%scope_load {struphy_profile_file} -n struphy_run -q
%scope_last struphy_run --include ^setup: -p

Other useful magics are %%scope_line for a short decorated function, %%scope_recursive for exploratory tracing when no regions exist, %scope_df for pandas analysis, %scope_compare for two notebook runs, and %scope_export for .prof or speedscope output. Recursive tracing is noisy and expensive, so use it to discover candidate functions rather than to measure production performance.

capture_full_output(
    profile_results.print_summary,
    title="Poisson profiling summary",
    include=[r"^setup:", r"^model\.integrate$", r"^prop: ", r"^solve:", r"^update_feec_variables$"],
)

from scope_profiler import plot_durations, plot_flame, plot_gantt

include_regions = [
    r"^setup:",
    r"^model\.integrate$",
    r"^prop: ",
    r"^solve:",
    r"^kernel: ",
    r"^update_feec_variables$",
]

duration_figure, _ = plot_durations(
    profile_results,
    include=include_regions,
    sort_by="total",
    top_n=12,
    stack_children=True,
    filepath=str(workdir / "durations_with_children.png"),
    return_fig=True,
)

gantt_figure, _ = plot_gantt(
    profile_results,
    include=include_regions,
    min_duration=1e-4,
    collapse_depth=4,
    filepath=str(workdir / "gantt.png"),
    return_fig=True,
)

flame_figure = plot_flame(
    profile_results,
    include=include_regions,
    filepath=str(workdir / "flame_graph.png"),
    return_fig=True,
)

The duration chart ranks regions by aggregate time and stacks self time with direct children. The Gantt chart shows the execution timeline and can expose idle time, serialization, or rank imbalance. The flame graph reconstructs nested calls and is useful for reading parent/child relationships between setup, model.integrate, propagators, solvers, and kernels. The figures are also saved under workdir, so the same workflow works in a headless batch job.

Cleanup

Remove the temporary directory when you no longer need the profiling files.

# shutil.rmtree(workdir)