[EPIC]: Bring-up and test coverage for PSO-based raytracing

[MarijnS95]

Companion to the inline-RT epic in #1258. That issue is scoped to RayQuery / TraceRayInline inside a regular compute shader; the test framework already treats it as a Compute pipeline that happens to bind an acceleration structure. PSO-based raytracing is a different beast — it introduces an entirely new pipeline kind, new shader stages, a shader binding table, and the DispatchRays command. The framework has none of that today, so this issue tracks both the bring-up work and the shader-visible test surface that becomes unlockable as bring-up progresses.

Reuses the AS YAML / abstraction layers from the inline-RT bring-up (#1214, #1232, #1245), so this work starts above that line.

Legend:

• Not started
• In progress 🏗️
• In review 👀
• Finished (merged in main)
• 👍 testable on top of the bring-up below
• 🏗️ requires additional framework work first

Bring-up PRs (foundation for everything below)

• Add ShaderPipelineKind::RayTracing, six new RT Stages, HitGroups / RayTracingPipelineConfig / ShaderBindingTable YAML schema, %dxc_target_lib substitution, the raytracing-pipeline lit feature, and a foundational XFAIL: * smoke test 👀
  • Add RayTracing pipeline kind, shader stages, and YAML schema #1270
• Vulkan RT pipeline + SBT + dispatchRays (drops Vulkan from the smoke test's XFAIL list) 👀
  • [VK] Add ray tracing pipeline, SBT, and DispatchRays bring-up #1273
• D3D12 state object + SBT + DispatchRays (drops DirectX from the smoke test's XFAIL list) 👀
  • [DX] Add ray tracing state object, SBT, and DispatchRays bring-up #1275
• Metal RT pipeline + SBT + dispatchRays via metal_irconverter (drops Metal from the smoke test's XFAIL list) 👀
  • [Metal] Add ray tracing pipeline, SBT, and DispatchRays bring-up #1281
• First PSO RT test batch: DispatchRays index/dimensions + SBT miss/hit-group routing 👀
  • Cover DispatchRays index/dimensions + SBT miss/hit-group routing #1277
• ClosestHit ray system values: barycentrics, PrimitiveIndex, WorldRay 👀
  • Cover ClosestHit ray system values: barycentrics, PrimitiveIndex, WorldRay #1278
• PSO-only RT surface: TraceRay recursion, RAY_FLAG_SKIP_CLOSEST_HIT_SHADER, callable shaders 👀
  • Cover PSO-only RT surface: recursion, RAY_FLAG_SKIP_CLOSEST_HIT_SHADER, callables #1279

Framework bring-up

Pipeline / stage model

• Add ShaderPipelineKind::RayTracing alongside Compute / TraditionalRaster / MeshShaderRaster in Support/Pipeline.h 👀
• Add new Stages: RayGeneration, Miss, ClosestHit, AnyHit, Intersection, Callable 👀
• Extend validatePipelineKind to require ≥1 raygen entry and at least one of {miss, hit-group} when the pipeline is RT 👀
• YAML mapping for the new stages and pipeline kind 👀

Shader compilation

• Add a lib_6_3 (or lib_6_5) target so clang-dxc / dxc emits a DXIL library with multiple entry points
• %dxc_target_lib substitution analogous to %dxc_target 👀
• Lit feature raytracing-pipeline (mirrors acceleration-structure) gated on the device feature + library-target support 👀

Hit groups, payloads, attributes, recursion config

• YAML HitGroups: list (Name, Type: Triangles|Procedural, ClosestHit, optional AnyHit, optional Intersection) 👀
• YAML RayTracingPipelineConfig: block (MaxTraceRecursionDepth, MaxPayloadSizeInBytes, MaxAttributeSizeInBytes, optional PipelineFlags) 👀
• Local root signatures (per shader-record root args) — at minimum a schema that lets a test record SBT-local constants 🏗️
• Validate payload/attr sizes against shader-declared sizes at YAML parse time

State Object / RT pipeline creation

• D3D12 backend: build a ID3D12StateObject from the YAML (DXIL library subobject, hit-group subobjects, global/local root signatures, raytracing pipeline + shader config subobjects)
• Vulkan backend: VkRayTracingPipelineKHR + vkGetRayTracingShaderGroupHandlesKHR, pipeline-library plumbing if needed; VK_KHR_ray_tracing_pipeline feature gate
• Metal backend: route through metal_irconverter for DXIL→AIR — it lowers DispatchRays into a compute dispatch backed by MTLVisibleFunctionTable / MTLIntersectionFunctionTable standing in for hit-groups, miss shaders, and callables. The backend work covers emitting the converter-expected compute dispatch and building those function tables. 👀 ([Metal] Add ray tracing pipeline, SBT, and DispatchRays bring-up #1281)

Shader binding table

• Abstract ShaderBindingTable resource owned by the RT pipeline
• YAML ShaderBindingTable: block describing raygen / miss / hit-group / callable records, each with ShaderName, optional local-root data 👀
• Per-backend SBT builders:
  • D3D12: identifier from ID3D12StateObjectProperties::GetShaderIdentifier, 32-byte aligned records, programmer-managed stride
  • Vulkan: VkStridedDeviceAddressRegionKHR, alignment from rayTracingPipelineProperties
  • Metal: visible function tables / intersection function tables
• Compute SBT record stride from declared local-root data size

DispatchRays command

• New RayTracingEncoder (or extend ComputeEncoder) with dispatchRays(width, height, depth, sbt)
• YAML dispatch parameters: RayTracingDispatch: { Width, Height, Depth } — currently reusing DispatchParameters.DispatchGroupCount as {Width, Height, Depth} for RT pipelines per Add RayTracing pipeline kind, shader stages, and YAML schema #1270; a dedicated RayTracingDispatch block can come later if useful.
• Resource-state / barrier handling for SBT and AS reads

Resource binding parity

• AS binding from raygen / miss / hit / callable stages (the inline-RT work only wires it up for compute)
• UAV / SRV / CBV / sampler tables from each new stage
• Verify cross-API root-signature layout matches between D3D12/Vulkan so the same DXIL works unmodified

Shader-observable spec surface

Each test asserts behavior visible from a raygen / miss / hit / callable shader. Coverage parallels the inline-RT epic where the same concept appears (e.g. ray flags), then adds the PSO-only surface (payloads, recursion, etc.).

TraceRay (host-side intrinsic)

• RayFlags parameter behaves the same as in RayQuery<> — re-run the same flag matrix from [EPIC]: Improve inline raytracing test coverage #1258 — 👍 once bring-up lands
• InstanceInclusionMask filtering — 👍
• RayContributionToHitGroupIndex selects the right hit-group record — 👍
• MultiplierForGeometryContributionToHitGroupIndex (per-geometry hit-group striding) — 🏗️ (multi-geometry BLAS)
• MissShaderIndex selects the right miss record — 👍
• RayDesc corners: TMin > 0, TMax clip, NaN/Inf direction — 👍
• Recursive TraceRay up to MaxTraceRecursionDepth — 👍
• Recursion beyond declared MaxTraceRecursionDepth is undefined; verify that the in-spec maximum value works — 👍

Raygen shader ([shader("raygeneration")])

• DispatchRaysIndex() matches the dispatch x/y/z lane — 👍
• DispatchRaysDimensions() echoes the host-side width/height/depth — 👍
• Multiple raygen entries, one selected per pipeline — 👍
• Writes to UAV indexed by DispatchRaysIndex — 👍
• Calls TraceRay and consumes the returned payload — 👍

Miss shader ([shader("miss")])

• Miss runs when no geometry is hit and payload is observable in raygen — 👍
• Multiple miss shaders, MissShaderIndex selects between them — 👍
• Miss-side ray-system-values: WorldRayOrigin, WorldRayDirection, RayTMin, RayTCurrent, RayFlags, DispatchRaysIndex — 👍

Closest-hit shader ([shader("closesthit")])

• Runs exactly once per accepted hit; payload mutation observable in raygen — 👍
• Hit-side ray-system-values: all of the miss-side ones plus InstanceIndex, InstanceID, PrimitiveIndex, GeometryIndex, HitKind, ObjectRayOrigin/Direction, ObjectToWorld3x4/4x3, WorldToObject3x4/4x3 — 👍
• BuiltInTriangleIntersectionAttributes.barycentrics at known points — 👍
• Hit-group routing: distinct closest-hit per hit-group, selected by RayContributionToHitGroupIndex + instance contribution — 👍
• Closest-hit calls TraceRay to spawn a secondary ray (recursion) — 👍

Any-hit shader ([shader("anyhit")])

• Any-hit fires for non-opaque geometry only — 🏗️ (non-opaque YAML)
• IgnoreHit() skips a candidate; closest-hit sees next hit — 🏗️
• AcceptHitAndEndSearch() ends traversal immediately — 🏗️
• Any-hit can mutate payload before IgnoreHit (and the mutation must be discarded per spec) — 🏗️
• Per-geometry NO_DUPLICATE_ANYHIT_INVOCATION — 🏗️ (geometry flag YAML)

Intersection shader ([shader("intersection")])

• ReportHit(t, kind, attrs) with custom hit-attribute struct — 🏗️ (procedural geometry)
• Custom HitKind value flows through to closest-hit / any-hit — 🏗️
• ReportHit returns true/false based on TMin/TMax + any-hit decision — 🏗️
• Intersection-side ray-system-values match the rest — 🏗️

Callable shader ([shader("callable")])

• CallShader(index, params) runs the right callable record — 👍
• Callable can be invoked from raygen, miss, closest-hit, callable — 👍
• DispatchRaysIndex / Dimensions visible inside callable — 👍

Payload + attribute mechanics

• Trivial payload (single uint) round-trip — 👍
• Large payload near MaxPayloadSizeInBytes — 👍
• [payload] qualifier in / out / inout semantics — 👍
• Custom hit attribute struct from an intersection shader — 🏗️
• Payload struct with mixed types (float/int/half) — 👍

Hit-group / SBT selection (shader-observable)

• InstanceContributionToHitGroupIndex selects per-instance hit group — 🏗️ (YAML field shared with [EPIC]: Improve inline raytracing test coverage #1258)
• RayContributionToHitGroupIndex per TraceRay call — 👍
• MultiplierForGeometryContributionToHitGroupIndex across multi-geom BLAS — 🏗️ (multi-geom YAML)
• Local-root constants in an SBT record are visible to the shader — 👍

Shader control flow / vendor edges

• Divergent rays per lane (different hit-groups invoked) — 👍
• Conditional TraceRay (lane mask) — 👍
• TraceRay with RAY_FLAG_SKIP_CLOSEST_HIT_SHADER (PSO-only flag) — 👍
• Mix of opaque and non-opaque BLAS in one pipeline — 🏗️
• Same ray flag matrix as [EPIC]: Improve inline raytracing test coverage #1258 for parity — 👍

Out of scope (intentionally)

• AS build flags / compaction / refit / clone / serialize — not shader-visible; same reasoning as [EPIC]: Improve inline raytracing test coverage #1258.
• DXR 1.2 / SER (ShaderExecutionReordering) — future epic once base PSO RT works on all three backends.
• Work-graph dispatched RT — separate feature.