Xe2 performance layer: XMX flash-attention recipes, managed SYCL JIT cache, on-device autotune

README.md

@@ -28,8 +28,9 @@ something useful before lunch.
 - **Auto-discovery** of every Intel GPU on the host (`Alchemist`, `Battlemage`,
   Lunar Lake iGPU). PCI device-ID table covers the common SKUs and falls back
   to OpenCL device-name parsing for the rest.
-- **Per-arch SYCL profiles** , env vars like `SYCL_CACHE_PERSISTENT=0` are
-  applied automatically, and known-bad ones (e.g. `GGML_SYCL_DISABLE_OPT`,
+- **Per-arch SYCL profiles** , env vars like `SYCL_CACHE_PERSISTENT` are
+  managed automatically (see the managed JIT cache below), and known-bad ones
+  (e.g. `GGML_SYCL_DISABLE_OPT`,
   `SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS`) are stripped from the
   inherited shell environment.
 - **Smart defaults** for `-ctx`, `--cache-type-k/v`, and `-ngl` based on the
@@ -109,6 +110,47 @@ curl http://127.0.0.1:11437/v1/chat/completions \
   cmake --build build --config Release -j
   ```
 - User in the `render` and `video` groups (`arc-llama doctor` will tell you).
+- For the XMX flash-attention prefill path (below), add `-DGGML_SYCL_DNN=ON`
+  (oneDNN ≥ 3.11.2, llama.cpp ≥ b10016 / 2026-07-15).
+
+## Performance: beyond stock settings
+
+arc-llama doesn't just avoid the Arc footguns — it applies optimizations no
+other Intel runtime ships yet:
+
+- **XMX flash-attention aware recipes.** llama.cpp's oneDNN SDPA path
+  (merged 2026-07-15) routes long prefills through the XMX systolic arrays —
+  up to **4.26× faster prefill at 80k ctx** on Battlemage — but only with an
+  **f16 KV cache**; quantized KV silently falls back to shader kernels. On
+  Xe2, arc-llama now defaults to f16 KV + `-fa on` + `-ub 1024 -b 2048`
+  whenever VRAM still allows ≥16k context, and drops back to q8_0 KV (more
+  context per byte) only on VRAM-tight setups. Every other tool leaves you
+  with the pre-XMX defaults — quantized KV that quietly disables the fastest
+  attention path Intel hardware has.
+- **Managed persistent SYCL JIT cache.** Battlemage + oneAPI 2026.0 SIGSEGVs
+  when `SYCL_CACHE_PERSISTENT=1` reads stale cache entries; everyone works
+  around it by disabling the cache — paying **~20 s of JIT recompilation on
+  every cold start, forever**. arc-llama instead gives each
+  (llama-server build, GPU driver) combination its own fingerprinted cache
+  dir, so stale entries can't exist, and arms a crash guard: if a server ever
+  dies during warm-up with the cache active, that cache is wiped, poisoned,
+  and the safe `=0` behaviour returns automatically. Net effect: the JIT cost
+  is paid once per llama.cpp upgrade instead of on every model swap.
+  Opt out with `jit_cache = "off"` under `[server]`.
+- **Multi-turn prompt reuse.** Non-SWA recipes get `--cache-reuse 256`, so
+  follow-up chat turns re-use the shared-prefix KV instead of re-prefilling
+  the whole conversation.
+- **`arc-llama tune`** — on-device autotuning. With `arc-llama serve`
+  running, `arc-llama tune <model>` sweeps the configs that actually matter
+  on Arc (q8 KV vs f16+FA, ubatch 512/1024/2048), measures real prefill +
+  generation through your own stack, scores each by modelled request latency,
+  and persists the winner into the model's recipe. Your card, your driver,
+  your build — not someone else's benchmark table.
+- **Slow-quant warnings.** Q8_0 *weight* quants hit a known-bad kernel path
+  on Xe2 (~22% of memory bandwidth vs Q4_K_M's ~55%,
+  [llama.cpp#21517](https://github.com/ggml-org/llama.cpp/issues/21517)) —
+  registering one warns you to grab Q4_K_M/Q6_K instead of leaving you to
+  wonder why generation crawls.
 
 ## Multi-GPU
 

src/arc_llama/arch.py

@@ -32,6 +32,19 @@ class ArchProfile:
     """Whether q8_0 K/V cache produces correct generation on this arch."""
     prefer_uniform_quants: bool = True
     """If true, recommend Q4_K_M over Unsloth Dynamic XL/UD variants."""
+    supports_xmx_fa: bool = False
+    """Whether the oneDNN XMX flash-attention prefill path (llama.cpp
+    PR #25222, merged 2026-07-15) is confirmed working on this arch.
+    The path requires an f16 K/V cache — quantized KV silently falls back
+    to the shader kernels, so recipes trade KV size for prefill speed."""
+    slow_weight_quants: tuple[str, ...] = ()
+    """Weight-quant tiers with known-bad kernel efficiency on this arch.
+    Used to warn at registration time, e.g. Q8_0 on Xe2 runs at ~22% of
+    memory bandwidth vs Q4_K_M's ~55% (ggml-org/llama.cpp#21517)."""
+    managed_jit_cache: bool = False
+    """If true, arc-llama replaces this profile's SYCL_CACHE_PERSISTENT=0
+    workaround with a managed per-binary cache dir (see sycl_cache.py),
+    recovering the ~20s cold-start JIT cost without the stale-cache SIGSEGV."""
 
 
 # ---------------------------------------------------------------------------
@@ -150,12 +163,22 @@ class ArchProfile:
     notes=[
         "Requires kernel 6.14+ and Mesa 24.x+ for stable `xe` driver.",
         "ReBAR REQUIRED — without it llama.cpp will fall back to slow paths.",
-        "First inference per cold start pays ~20s of SYCL JIT compile.",
+        "First inference per cold start pays ~20s of SYCL JIT compile "
+        "(recovered automatically when the managed JIT cache is active).",
         "q8_0 K/V cache works correctly but on some builds underutilises memory "
         "bandwidth on dense models. Verify perf if you care; correctness is OK.",
+        "llama.cpp builds with GGML_SYCL_DNN=ON route long prefills through the "
+        "XMX engines when flash attention is on and the KV cache is f16 — up to "
+        "4.26x prefill at 80k ctx (PR #25222). arc-llama recipes prefer f16 KV "
+        "on this arch when VRAM allows.",
+        "Avoid Q8_0 *weight* quants here — Q4_K_M/Q4_K_S/Q6_K run 4-5x faster "
+        "(llama.cpp issue #21517).",
     ],
     safe_kv_q8=True,
     prefer_uniform_quants=True,
+    supports_xmx_fa=True,
+    slow_weight_quants=("Q8_0",),
+    managed_jit_cache=True,
 )
 
 LUNAR_LAKE_PROFILE = ArchProfile(
@@ -177,6 +200,12 @@ class ArchProfile:
     ],
     safe_kv_q8=True,
     prefer_uniform_quants=True,
+    # Xe2-LPG carries the same XMX engines and JIT-cache behaviour as
+    # Battlemage; the shared-memory VRAM budget is what differs, and that is
+    # already handled by ctx sizing.
+    supports_xmx_fa=True,
+    slow_weight_quants=("Q8_0",),
+    managed_jit_cache=True,
 )
 
 UNKNOWN_PROFILE = ArchProfile(

src/arc_llama/benchmark.py

@@ -366,6 +366,189 @@ async def benchmark_sweep(
     return results
 
 
+# ------------------------------------------------------------------
+# Autotune
+# ------------------------------------------------------------------
+
+TUNE_PROMPT_TOKENS = 2048
+TUNE_GEN_TOKENS = 128
+"""Autotune workload shape: long enough prefill to expose the XMX/oneDNN
+flash-attention path (which only engages ≥32-token query batches and pays
+off at scale), plus a chat-sized generation burst."""
+
+
+@dataclass
+class TuneCandidate:
+    """One recipe variant the autotuner will measure."""
+    label: str
+    edit: dict[str, Any]
+
+
+@dataclass
+class TuneOutcome:
+    candidate: TuneCandidate
+    result: BenchmarkResult
+
+    @property
+    def request_seconds(self) -> float | None:
+        """Modelled wall time of one real request: prefill + generation.
+
+        This is the scoring metric — it weighs prompt and generation speed by
+        how much time each actually costs, instead of an arbitrary blend.
+        """
+        pp = self.result.prompt_eval_tok_s
+        gen = self.result.generation_tok_s
+        if not pp or not gen:
+            return None
+        return self.result.prompt_tokens / pp + self.result.gen_tokens / gen
+
+
+def build_tune_candidates(arch_value: str, current_recipe: dict[str, Any]) -> list[TuneCandidate]:
+    """Candidate grid for one model on one arch.
+
+    Kept deliberately small — each candidate costs a full model reload plus
+    JIT warm-up, so we probe the configurations that actually move the needle
+    on Arc rather than a blind grid:
+      * q8_0 KV without forced FA (the pre-XMX-era default, still best when
+        the quantized-KV shader path happens to win on a given build);
+      * f16 KV + -fa on across ubatch 512/1024/2048 (the oneDNN XMX prefill
+        path needs f16 KV; ubatch controls how well-fed the systolic arrays are).
+    """
+    from arc_llama.arch import Arch, profile_for
+    try:
+        profile = profile_for(Arch(arch_value))
+    except ValueError:
+        profile = profile_for(Arch.UNKNOWN)
+
+    cands = [TuneCandidate(
+        label="current",
+        edit={
+            k: current_recipe[k]
+            for k in ("cache_type_k", "cache_type_v", "ubatch_size", "batch_size", "flash_attn")
+            if k in current_recipe
+        } or {"flash_attn": None},
+    )]
+    cands.append(TuneCandidate(
+        label="q8-kv",
+        edit={"cache_type_k": "q8_0", "cache_type_v": "q8_0",
+              "flash_attn": None, "ubatch_size": 512, "batch_size": None},
+    ))
+    ubatches = [512, 1024, 2048] if profile.supports_xmx_fa else [512, 1024]
+    for ub in ubatches:
+        cands.append(TuneCandidate(
+            label=f"f16-fa-ub{ub}",
+            edit={"cache_type_k": "f16", "cache_type_v": "f16",
+                  "flash_attn": "on", "ubatch_size": ub, "batch_size": max(2048, ub)},
+        ))
+    return cands
+
+
+async def autotune_model(
+    server_url: str,
+    model_name: str,
+    *,
+    prompt_tokens: int = TUNE_PROMPT_TOKENS,
+    gen_tokens: int = TUNE_GEN_TOKENS,
+    apply_best: bool = True,
+    cfg: Config | None = None,
+) -> tuple[list[TuneOutcome], TuneOutcome | None]:
+    """Measure each candidate on the real hardware and keep the fastest.
+
+    Every candidate goes through the running arc-llama server (correct SYCL
+    env, arch profile, managed JIT cache), gets a full reload, warm-up, and
+    prefill+generation measurement. Winner = lowest modelled request time.
+    Applies the winner to the model's recipe unless apply_best=False, in
+    which case the original recipe is restored.
+    """
+    if cfg is None:
+        cfg = load_config()
+    model = cfg.find_model(model_name)
+    if model is None:
+        raise ValueError(f"Model '{model_name}' not found in config")
+    gpu = cfg.find_gpu(model.gpu_pci_slot)
+    arch_value = gpu.arch if gpu else "unknown"
+    original_recipe = dict(model.recipe or {})
+
+    candidates = build_tune_candidates(arch_value, original_recipe)
+    outcomes: list[TuneOutcome] = []
+    # MTP models pin ubatch to 8 — sweeping prefill batches would fight the
+    # SSM compute-buffer constraint, so only the current config is measured.
+    if original_recipe.get("spec_type") == "draft-mtp":
+        log.warning("%s uses draft-mtp; skipping batch/FA sweep, keeping current recipe", model_name)
+        candidates = candidates[:1]
+
+    async with httpx.AsyncClient(base_url=server_url, timeout=600.0) as client:
+        for cand in candidates:
+            log.info("tuning %s: %s ...", model_name, cand.label)
+            r = await client.post(f"/admin/models/{model_name}/edit", json=cand.edit)
+            if r.status_code != 200:
+                outcomes.append(TuneOutcome(cand, BenchmarkResult(
+                    model=model_name, ctx=0,
+                    cache_type_k=str(cand.edit.get("cache_type_k", "?")),
+                    cache_type_v=str(cand.edit.get("cache_type_v", "?")),
+                    prompt_tokens=prompt_tokens, gen_tokens=gen_tokens,
+                    error=f"edit failed: {r.status_code} {r.text}",
+                )))
+                continue
+            res = await benchmark_model(
+                server_url, model_name,
+                prompt_tokens=prompt_tokens, gen_tokens=gen_tokens,
+                load=True, cfg=cfg,
+            )
+            outcomes.append(TuneOutcome(cand, res))
+
+        scored = [o for o in outcomes if o.request_seconds is not None and not o.result.error]
+        best = min(scored, key=lambda o: o.request_seconds) if scored else None
+
+        # Land on the winner (or roll back). The edit endpoint persists to
+        # config.toml, so whatever we set last is what survives.
+        final_edit = best.candidate.edit if (best and apply_best) else {
+            k: original_recipe.get(k)
+            for k in ("cache_type_k", "cache_type_v", "ubatch_size", "batch_size", "flash_attn")
+        }
+        final_edit = {k: v for k, v in final_edit.items() if k in (
+            "cache_type_k", "cache_type_v", "ubatch_size", "batch_size", "flash_attn",
+        )}
+        # cache_type must never be null; drop absent keys instead.
+        for k in ("cache_type_k", "cache_type_v"):
+            if final_edit.get(k) is None:
+                final_edit.pop(k, None)
+        if final_edit:
+            await client.post(f"/admin/models/{model_name}/edit", json=final_edit)
+
+    return outcomes, best
+
+
+def print_tune_table(outcomes: list[TuneOutcome], best: TuneOutcome | None) -> None:
+    from rich.console import Console
+    from rich.table import Table
+
+    console = Console()
+    table = Table(title="Autotune")
+    table.add_column("candidate")
+    table.add_column("KV")
+    table.add_column("Prompt-eval")
+    table.add_column("Generation")
+    table.add_column("2k-req time")
+    table.add_column("")
+    for o in outcomes:
+        r = o.result
+        if r.error:
+            table.add_row(o.candidate.label, f"{r.cache_type_k}/{r.cache_type_v}",
+                          "[red]error[/red]", "", "", "")
+            continue
+        secs = o.request_seconds
+        table.add_row(
+            o.candidate.label,
+            f"{r.cache_type_k}/{r.cache_type_v}",
+            _fmt_speed(r.prompt_eval_tok_s),
+            _fmt_speed(r.generation_tok_s),
+            f"{secs:.2f} s" if secs is not None else "—",
+            "◀ winner" if best is o else "",
+        )
+    console.print(table)
+
+
 # ------------------------------------------------------------------
 # Formatting
 # ------------------------------------------------------------------

src/arc_llama/cli.py

@@ -9,6 +9,7 @@
   arc-llama add        Register a model — local file or HF download.
   arc-llama remove     Remove a model from the config.
   arc-llama serve      Run the OpenAI-compatible router.
+  arc-llama tune       Autotune a model's recipe on the actual hardware.
   arc-llama tui        Launch the terminal UI.
   arc-llama systemd    Print a systemd --user service unit for `arc-llama serve`.
 
@@ -615,6 +616,72 @@ def _on_signal(signum: int, _frame) -> None:  # noqa: ANN001
     uvicorn.run(app, host=cfg.server.host, port=cfg.server.port, log_level="info")
 
 
+# ===========================================================================
+# tune
+# ===========================================================================
+
+@cli.command("tune")
+@click.argument("model")
+@click.option("--server-url", default=None,
+              help="Base URL of the running `arc-llama serve` (default: from config).")
+@click.option("--prompt-tokens", type=int, default=None,
+              help="Prefill size to benchmark (default: 2048).")
+@click.option("--gen-tokens", type=int, default=None,
+              help="Generation burst to benchmark (default: 128).")
+@click.option("--no-apply", is_flag=True,
+              help="Measure only; restore the original recipe afterwards.")
+@click.pass_context
+def tune_cmd(
+    ctx: click.Context,
+    model: str,
+    server_url: str | None,
+    prompt_tokens: int | None,
+    gen_tokens: int | None,
+    no_apply: bool,
+) -> None:
+    """Autotune a model's recipe on the actual hardware.
+
+    Sweeps the configurations that matter on Arc — q8_0 KV vs f16 KV with
+    flash attention forced on (the oneDNN XMX prefill path), across ubatch
+    sizes — through the running `arc-llama serve`, then persists the fastest.
+    Requires the server to be running; each candidate costs a model reload.
+    """
+    import asyncio
+
+    from arc_llama.benchmark import (
+        TUNE_GEN_TOKENS,
+        TUNE_PROMPT_TOKENS,
+        autotune_model,
+        print_tune_table,
+    )
+    cfg = load_config(ctx.obj["config_path"])
+    url = server_url or f"http://{cfg.server.host}:{cfg.server.port}"
+    try:
+        outcomes, best = asyncio.run(autotune_model(
+            url, model,
+            prompt_tokens=prompt_tokens or TUNE_PROMPT_TOKENS,
+            gen_tokens=gen_tokens or TUNE_GEN_TOKENS,
+            apply_best=not no_apply,
+            cfg=cfg,
+        ))
+    except ValueError as e:
+        console.print(f"[red]{e}[/red]")
+        sys.exit(1)
+    except Exception as e:
+        console.print(f"[red]tune failed: {e}[/red]")
+        console.print("Is `arc-llama serve` running at "
+                      f"{url}? Start it first — tuning runs through the live server.")
+        sys.exit(1)
+    print_tune_table(outcomes, best)
+    if best is None:
+        console.print("[yellow]No candidate produced a valid measurement.[/yellow]")
+        sys.exit(1)
+    if no_apply:
+        console.print(f"Winner: [bold]{best.candidate.label}[/bold] (original recipe restored).")
+    else:
+        console.print(f"Applied winner: [bold]{best.candidate.label}[/bold] → {best.candidate.edit}")
+
+
 # ===========================================================================
 # mtp-info
 # ===========================================================================