feat(exgentic-runner): in-cluster execution support

[yoavkatz]

Summary

Add full in-cluster execution support for the exgentic A2A runner so benchmarks can be run as Kubernetes Jobs without any local machine involvement. Also introduces an end-to-end test script, several correctness fixes for parallel and in-cluster runs, and documentation updates.

---

New files

e2e-test.sh

Orchestrates a full end-to-end test across all benchmarks (gsm8k, tau2, appworld) by calling deploy-and-evaluate.sh once per benchmark. Key features:

• Sequential mode (default): stops on first failure; marks remaining benchmarks as SKIP
• --parallel-jobs: runs all benchmarks concurrently; each gets unique OTEL-collector and Prometheus local ports so port-forwards do not collide
• --in-cluster: submits a kubectl Job per benchmark derived from k8s/job.yaml, streams logs, checks job success status
• --tasks N / --parallel-sessions N: forwarded as MAX_TASKS / MAX_PARALLEL_SESSIONS env vars
• All unrecognised flags are passed through verbatim to deploy-and-evaluate.sh
• Prints and writes e2e-results.md with a summary table: benchmark, status (PASS/FAIL/SKIP), eval success rate, avg latency, failure count

libsh/urls.sh

Shared URL helper library sourced by all deploy/evaluate scripts. Each function returns the correct URL depending on whether KUBERNETES_SERVICE_HOST is set (in-cluster cluster-DNS) or not (developer laptop localtest.me):

• kagenti_api_url — Kagenti backend
• keycloak_api_url — Keycloak
• tool_http_url / tool_k8s_url — MCP server
• agent_http_url — A2A agent
• mcp_gateway_url — MCP Gateway
• otel_collector_url — OTEL Collector (HTTP/protobuf port 8335 in-cluster)
• prometheus_url — Prometheus

k8s/job.yaml

Kubernetes Job template for in-cluster runner execution. Reads credentials from secrets (openai-secret, kagenti-test-user, hf-secret), uses the default ServiceAccount (no kubectl RBAC needed), and has ttlSecondsAfterFinished: 3600 / backoffLimit: 0.

sync-image-to-cluster.sh

Extracted helper (previously inlined in deploy scripts) that loads a locally-built container image into the kind cluster. Only needed when --local-image is passed; normal runs pull from the GHCR registry.

update-secrets.sh

Convenience script to create/patch openai-secret and hf-secret in a target namespace via kubectl. Run from a local machine before submitting Jobs.

Dockerfile

Container image definition for the runner itself, used to produce ghcr.io/exgentic/runner:latest.

---

Changed files

deploy-benchmark.sh / deploy-agent.sh

• Remote registry by default: scripts now pull ghcr.io/exgentic/<image>:latest directly; local image sync is only triggered with --local-image
• URL helpers: hardcoded localtest.me URLs replaced with libsh/urls.sh calls so the same scripts work on a dev laptop and inside the cluster
• Env var precedence: default values now use ${VAR:-default} so secrets injected via Kubernetes Job env vars are not overwritten by .env file loading
• Removed: inline container-runtime detection and docker pull / podman tag logic (moved to sync-image-to-cluster.sh)

evaluate-benchmark.sh

• No port-forwarding for MCP/agent: services are reached via HTTP routes (or cluster DNS in-cluster); only the OTEL Collector still uses a port-forward on a dev laptop
• MLflow enabled by default: --mlflow flag replaced by --disable-mlflow; tracing is on unless explicitly turned off
• In-cluster guard: entire kubectl context check block is skipped when KUBERNETES_SERVICE_HOST is set — avoids hangs from interactive prompts inside a pod
• Env var precedence: .env file is read with ${VAR:-} guarding so CLI/env overrides are respected
• Parallel port offsets: OTEL_COLLECTOR_LOCAL_PORT and PROMETHEUS_LOCAL_PORT default from environment so e2e-test.sh can inject unique values per concurrent benchmark
• HTTP health checks: replaced kubectl wait pod-readiness checks with polling wait_for_url against the service HTTP endpoints

deploy-and-evaluate.sh

• Propagates --disable-mlflow (was --mlflow) to evaluate-benchmark.sh
• Passes through MAX_TASKS / MAX_PARALLEL_SESSIONS from environment

exgentic_a2a_runner/otel.py

• Detects KUBERNETES_SERVICE_HOST to switch the OTEL exporter from gRPC (localhost:4317) to HTTP/protobuf (otel-collector…:8335) — the OTEL Collector in the Kagenti cluster only exposes the HTTP/protobuf port inside the cluster

AGENTS.md

• Added branching policy: always create a branch before committing; never commit directly to main
• Added DCO sign-off requirement (git commit -s)

README.md

• Removed image build steps from the Getting Started section (images are pre-built in GHCR)
• Added In-Cluster Execution section with step-by-step instructions (secrets setup → configure job → submit → view results → iterate with local image)
• Added E2E Test Script section documenting e2e-test.sh usage, flags, and the results table format
• Updated all --mlflow references to --disable-mlflow

---

Bug fixes

Commit	Fix
223b5cd	Preserve benchmark exit status in parallel runs — subshell exit codes were lost when run_benchmark was backgrounded
b5439b0	Respect CLI/env overrides over .env file — .env was overwriting already-exported vars
1b1a7e1	Assign unique port-forward ports per benchmark in parallel mode — concurrent e2e-test.sh runs were binding the same local ports
2a4c0f3	Use in-cluster Kubernetes DNS URL for MCP service in the Job spec instead of the external localtest.me route
953b221	Use HTTP/protobuf OTEL exporter in-cluster; rename --mlflow → --disable-mlflow
aa9251e	Replace kubectl wait with HTTP health checks so scripts work without a kubeconfig