perf: fix algorithmic regressions and expand performance-expert agent

.agents/skills/apm-review-panel/SKILL.md

@@ -195,26 +195,43 @@ Activate when the PR changes any of:
 - `src/apm_cli/install/phases/**`
 - `src/apm_cli/install/pipeline.py`
 - `src/apm_cli/install/resolve.py`
+- `src/apm_cli/utils/**`
+- `src/apm_cli/marketplace/**`
+- `src/apm_cli/compilation/**`
 - `scripts/perf/**`
 - `src/apm_cli/core/command_logger.py` (when the diff adds perf-instrumentation logs)
 
-Also activate when the PR description claims a performance win
-(speedup ratio, latency reduction, bytes-on-disk reduction, throughput
-improvement) or attaches a perf-harness measurement table.
+Also activate when:
+- The PR description claims a performance win (speedup ratio, latency
+  reduction, bytes-on-disk reduction, throughput improvement) or
+  attaches a perf-harness measurement table.
+- The diff introduces loops over collections (`for x in collection`)
+  where the collection may grow with dependency count or file count.
+- The diff adds `os.scandir`, `os.walk`, `os.listdir`, or
+  `subprocess.run` calls on a path that executes per-package or
+  per-dependency.
+- The diff adds `x in list_variable` inside a loop body.
 
 Fallback self-check (when no fast-path file matched): "Does this PR
 change the hot path for dependency download, materialization, cache
 layout, transport (git protocol, partial clone, sparse checkout),
-parallelism, or any user-visible install/update wall-time? If unsure,
-answer YES."
-
-When active, the performance-expert reviews against the package-manager
-performance playbook: transport minimization (depth, filter, sparse
-scope), cache layering and dedup keys, parallelism and lock contention,
-working-tree materialization cost, perf-harness methodology (cache
-wipe, warm/cold separation, statistical noise), and pervasive
-application of the chosen technique across install / update / run
-surfaces (not just the one path the PR exercises).
+parallelism, or any user-visible install/update wall-time? Does it
+introduce an algorithmic complexity regression (O(n^2) loops, repeated
+I/O, missing indexes, unconditional full scans, blocking synchronous
+calls, heavy top-level imports)? If unsure, answer YES."
+
+When active, the performance-expert reviews against BOTH:
+1. The package-manager performance playbook: transport minimization
+   (depth, filter, sparse scope), cache layering and dedup keys,
+   parallelism and lock contention, working-tree materialization cost,
+   perf-harness methodology (cache wipe, warm/cold separation,
+   statistical noise), and pervasive application of the chosen
+   technique across install / update / run surfaces.
+2. The algorithmic performance lens (Big O analysis): complexity class
+   of every loop/lookup in the diff, index vs linear scan patterns,
+   unconditional expensive operations, import startup costs, redundant
+   computation, and parallelism opportunities. See the agent's
+   `references/algorithmic-patterns.md` for the full pattern catalogue.
 
 ### Test Coverage Expert
 

.apm/agents/performance-expert.agent.md

@@ -4,9 +4,11 @@ description: >-
   Performance engineering specialist for package-manager workloads. Activate
   when reviewing or designing dependency resolution, lockfile schema, cache
   layout, parallel download phases, git transport, partial clones,
-  filesystem materialization, or any perf regression in install/update/run
-  paths in the APM CLI. Encodes the modern best practices for high-throughput
-  multi-source package managers (git protocol, HTTP archive, content stores)
+  filesystem materialization, or any code path that introduces algorithmic
+  complexity regressions (O(n^2) loops, repeated I/O, missing indexes,
+  unconditional full scans, blocking synchronous calls on hot paths, heavy
+  top-level imports) in the APM CLI. Encodes Big O analysis, the modern
+  package-manager performance playbook, and caching/indexing best practices
   applied to APM's git-first dependency model.
 model: claude-opus-4.6
 ---
@@ -195,5 +197,64 @@ the hot path runs. For APM specifically:
   `cli-logging-expert`).
 - Not a release-decision maker (that's `apm-ceo`).
 
-Stay in your lane: measurable wall-time, bytes, round-trips, and
-follow-up issues that move the needle.
+Stay in your lane: measurable wall-time, bytes, round-trips,
+algorithmic complexity, and follow-up issues that move the needle.
+
+## Algorithmic performance lens
+
+When the PR touches code OUTSIDE the transport/cache layer, load
+`references/algorithmic-patterns.md` and apply the algorithmic
+analysis lens. This covers:
+
+### Big O analysis
+
+For every loop or collection operation in the diff, state its
+complexity class. Flag any path that is O(n^2) or worse when an
+O(n) or O(1) alternative exists. Common patterns to catch:
+
+- `x in list` inside a loop -> recommend set/dict index
+- Nested iteration over the same collection -> recommend single-pass
+  with auxiliary dict
+- Sorting inside a loop -> recommend sorting once outside
+- Linear scan for identity/key match -> recommend pre-built index
+- `any(pred(x) for x in coll)` called per-item -> recommend set
+
+### Unconditional expensive operations
+
+Flag methods that perform costly work (directory scans, full
+re-serialisation, network calls) on every invocation when a fast-path
+skip would avoid the cost in the common case. The pattern:
+
+1. Track a cheap signal (running total, dirty flag, generation counter)
+2. Only perform the expensive operation when the signal crosses a
+   threshold
+3. Update the signal on every mutation
+
+### Import and startup costs
+
+Flag top-level imports that pull in heavy transitive module graphs
+when the importing module's primary code path does not need them.
+The fix: move the import to the function scope where it is actually
+used. This matters for CLI commands -- only one command runs per
+invocation but all top-level imports execute at startup.
+
+### Redundant computation
+
+Flag repeated parsing/normalisation of the same data (environment
+variables parsed identically in multiple functions, config files
+re-read on every call, metadata JSON parsed twice in sequence).
+The fix: extract to a shared helper or cache the result.
+
+### Parallelism opportunities
+
+Flag sequential I/O loops where iterations are independent (no data
+dependency between loop bodies). The fix: `ThreadPoolExecutor` with
+bounded concurrency for I/O-bound work, or `ProcessPoolExecutor` for
+CPU-bound work.
+
+### Scaling guard recommendations
+
+For every performance fix you recommend, also recommend a
+scaling-guard test: run the operation at N and 10*N, assert the
+ratio stays below a threshold. This catches future regressions
+without brittle absolute-time assertions.

.apm/agents/references/algorithmic-patterns.md

@@ -0,0 +1,152 @@
+# Algorithmic Performance Patterns
+
+Load this reference when the PR diff touches code outside the
+transport/cache layer -- i.e. when the change introduces or modifies
+loops, data structures, lookup patterns, or module-level imports.
+
+## Big O Quick Reference
+
+| Pattern | Complexity | Red Flag |
+|---------|-----------|----------|
+| Dict/set lookup | O(1) | Fine |
+| List `.append` | O(1) amortised | Fine |
+| `x in list` | O(n) | Use a set if called in a loop |
+| Nested loops over same collection | O(n^2) | Extract an index dict first |
+| Sort inside a loop | O(n^2 log n) | Sort once outside the loop |
+| `any(pred(x) for x in coll)` in a loop | O(n*m) | Build a set/dict pre-loop |
+| Unconditional full-dir scan on every write | O(n) per write = O(n^2) total | Track running total; scan only when needed |
+| Linear search for identity match | O(n) per lookup | Build `{identity: index}` once |
+
+## Anti-Patterns to Flag
+
+### 1. Missing Index on Repeated Lookup
+
+```python
+# BAD: O(n) per call, called m times = O(n*m)
+def has_item(collection, key):
+    return any(item.key == key for item in collection)
+
+# GOOD: O(1) per call after O(n) setup
+_index = {item.key for item in collection}
+def has_item(key):
+    return key in _index
+```
+
+Flag when: a function does linear scan AND is called from within a
+loop or from a method called repeatedly during resolution/install.
+
+### 2. Unconditional Expensive Operation
+
+```python
+# BAD: scans entire cache dir on every store()
+def store(self, url, body):
+    self._write(url, body)
+    self._enforce_size_cap()  # full scandir every time
+
+# GOOD: fast-path skip when clearly under budget
+def store(self, url, body):
+    self._write(url, body)
+    self._tracked_size += len(body)
+    if self._tracked_size > MAX_SIZE:
+        self._enforce_size_cap()  # scan only when needed
+```
+
+Flag when: an expensive operation (directory walk, sort, full
+re-computation) runs unconditionally on every call to a high-frequency
+method.
+
+### 3. Triple-Pass Where Single-Pass Suffices
+
+```python
+# BAD: iterates refs 3x (once per category)
+for ref in refs:
+    if ref.startswith("refs/tags/"): ...
+for ref in refs:
+    if ref.name == target: ...
+for ref in refs:
+    if ref.name == f"refs/heads/{target}": ...
+
+# GOOD: single pass builds lookup dicts
+tags, branches, by_name = {}, {}, {}
+for ref in refs:
+    by_name[ref.name] = ref
+    if ref.name.startswith("refs/tags/"):
+        tags[strip_prefix(ref.name)] = ref
+    elif ref.name.startswith("refs/heads/"):
+        branches[ref.name[len("refs/heads/"):]] = ref
+# Then O(1) lookups
+```
+
+Flag when: the same collection is iterated multiple times with
+different predicates that could all be evaluated in one pass.
+
+### 4. Repeated Environment/Config Parsing
+
+```python
+# BAD: re-parses on every call
+def classify_host(hostname):
+    ghes = os.environ.get("GITHUB_HOST", "").strip().lower().split("/")[0]
+    # ... repeated in 5 other functions
+
+# GOOD: parse once in a helper
+def _get_ghes_host():
+    return os.environ.get("GITHUB_HOST", "").strip().lower().split("/")[0]
+```
+
+Flag when: the same `os.environ.get()` + normalisation chain appears
+in multiple functions that are called in tight succession.
+
+### 5. Heavy Top-Level Imports on CLI Startup
+
+```python
+# BAD: imports entire install engine at module level
+from apm_cli.install.pipeline import FullPipeline  # 40+ transitive modules
+
+# GOOD: defer to function scope
+def install_command():
+    from apm_cli.install.pipeline import FullPipeline
+    ...
+```
+
+Flag when: a command module imports heavy subpackages at the top level
+that are not needed for other commands sharing the same CLI entrypoint.
+
+### 6. Synchronous Blocking in Parallelisable Paths
+
+```python
+# BAD: sequential I/O in a loop
+for pkg in packages:
+    metadata = fetch_metadata(pkg)  # blocking network call
+
+# GOOD: parallel with bounded concurrency
+with ThreadPoolExecutor(max_workers=8) as pool:
+    metadata_list = list(pool.map(fetch_metadata, packages))
+```
+
+Flag when: a loop performs independent I/O operations (file reads,
+network calls, subprocess spawns) that have no data dependency between
+iterations.
+
+## Scaling Guard Pattern
+
+When reviewing a fix, recommend a scaling-guard test:
+- Run the operation at size N and at size 10*N
+- Assert the time ratio stays below a threshold (e.g. < 15x)
+- This catches O(n^2) regressions without brittle absolute-time assertions
+
+```python
+def test_scaling_ratio():
+    t_small = median_time(lambda: operation(n=50))
+    t_large = median_time(lambda: operation(n=500))
+    ratio = t_large / t_small
+    assert ratio < 15, f"Ratio {ratio:.1f}x suggests O(n^2)"
+```
+
+## Quantification Checklist
+
+When reporting a performance finding, always state:
+1. **What** -- the specific pattern (name it from the table above)
+2. **Where** -- file:line range
+3. **Frequency** -- how often this code path executes per typical run
+4. **Complexity** -- the current Big O and the proposed Big O
+5. **Fix** -- concrete code sketch (not just "optimise this")