Skip to content

refactor, unify filter pipeline scheduling around FAPL context #287

Description

@steven-varga

Background

The current multithreaded filter path has a split policy model:

  • FAPL owns h5::threads{N} / h5::backpressure{N} and the worker pool.
  • DAPL owns h5::high_throughput and a per-dataset pipeline opt-in.
  • pool_pipeline_t still uses std::queue/future scheduling and per-chunk heap buffers, while the repo also contains H5Qall.hpp Vyukov queues and ring/arena machinery.

This makes the mental model harder than it needs to be for full async I/O: HDF5 calls must remain serialized per file, while filter work can run in parallel.

Goal

Introduce a file-level pipeline context/policy surface so users can express multithreaded read/write filter-chain behavior at FAPL scope, with DAPL acting as a per-dataset policy view rather than a separate execution universe.

Target user shape:

h5::fapl_t fapl = h5::pipeline::multithreaded{
    h5::threads{8},
    h5::arena{512_MiB},
    h5::backpressure{64}
};

Scope

  • Add the public FAPL-level pipeline policy syntax.
  • Preserve existing h5::threads{N} / h5::backpressure{N} behavior.
  • Keep h5::high_throughput working as the dataset opt-in / compatibility path.
  • Centralize the policy so future Vyukov queue + arena scheduling can hang from the same file-level context.
  • Add focused tests for syntax, policy resolution, and compatibility.

Non-goals

  • Do not rewrite the whole filter engine in one pass.
  • Do not expose low-level Vyukov queue topology as the default API.
  • Do not make concurrent HDF5 C-API calls from worker threads.

Acceptance Criteria

  • h5::pipeline::multithreaded{h5::threads{N}, h5::arena{...}, h5::backpressure{M}} compiles as a FAPL policy.
  • Existing h5::threads{N} tests continue to pass.
  • The policy preserves the invariant: HDF5 calls stay serialized; CPU filter work remains the parallelizable portion.
  • Existing high-throughput read/write tests remain green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions