chore(scripts): preserve 2026-07 MCP catalog sweep tooling + probe ledger

scripts/mcp-probe/README.md

@@ -0,0 +1,69 @@
+# TAS Native MCP catalog — 2026-07 sweep tooling + probe ledger
+
+Working set from the batch-3 catalog expansion
+([tembo/agent-studio#310](https://github.com/tembo/agent-studio/issues/310),
+PR [#313](https://github.com/tembo/agent-studio/pull/313)). Preserved so
+batch 4 doesn't start from scratch. All probes ran unauthenticated from a
+Tembo sandbox on **2026-07-18**.
+
+## Files
+
+| File | What it is |
+| --- | --- |
+| `probe.sh` | Shallow probe of one candidate: MCP `initialize` POST (expects 401 + `WWW-Authenticate` for OAuth servers) → `/.well-known/oauth-protected-resource` (path-aware, then origin root) → auth-server metadata → DCR check. Emits one TSV row. |
+| `probe2.sh` | Deep probe: emits one JSON object per provider with every OAuth origin the TAS catalog + Rust allowlist need (`authorization_endpoint`, `token_endpoint`, `registration_endpoint` origins). |
+| `driver-shallow.py` / `driver-deep.py` | Concurrent drivers that fan the probes over a candidate list. |
+| `generate.mjs` | Generates the `mcp-providers.ts` catalog entries (union member, entry object, category wiring) and the Rust allowlist tuples from the deep-probe JSON + a hand-curated metadata table (displayName, auth mode, category, caveat notes). This is the expensive-to-recreate piece. |
+| `fetch-art.mjs` | Logo fetcher: tries vendor favicon/apple-touch/known CDN paths, falls back to Google s2 favicons, md5-compares against the s2 "default globe" to reject placeholders, validates PNG/ICO/SVG/JPG magic bytes. |
+| `probe-results-2026-07-18.tsv` | **Full shallow-probe ledger: all 216 candidates**, including the ~80 that did not make the catalog. Columns: slug, MCP URL, `initialize` status, protected-resource status (+ which URL variant hit), auth server, DCR, notes. |
+
+## Reading the ledger
+
+- `200/…` or `401 + protected-resource 200` → live hosted server (candidates for the catalog).
+- `401/401` or `401/404` with no metadata → auth-gated, discovery inconclusive — re-probe candidates.
+- `404/404`, `405`, `ERR` → no hosted MCP server at that URL as of 2026-07-18 — the negative-result baseline for the next sweep (probe URLs are the best-known guess from registry/directory/community lists, so a vendor may still launch elsewhere).
+
+## Reproducing
+
+```bash
+cd scripts/mcp-probe
+mkdir -p /tmp/mcp-probe                # probe.sh scratch dir
+./probe.sh <slug> <mcp_url>            # one TSV row to stdout
+./probe2.sh <slug> <mcp_url>           # one JSON object to stdout
+node generate.mjs                      # reads deep/*.json, writes gen-*.txt fragments
+```
+
+Methodology is also documented in the batch-3 header of
+`web/src/lib/mcp-providers.ts`.
+
+## Outcomes recovered from the session transcript (not in the TSV)
+
+The TSV covers the `probe.sh` runs. A second, ad-hoc probe pass (the Python
+drivers) recorded some outcomes only in the research session transcript;
+recovered here so they aren't lost:
+
+- **Registry sweep scale:** official MCP registry swept in full — 542 pages,
+  54,191 entries, 8,424 active remote servers — before filtering to the
+  candidate set.
+- **Live, anonymous `initialize` OK (no-auth candidates):** InVideo
+  (`mcp.invideo.io/mcp`).
+- **Parked with reasons:** Pipedream
+  (`remote.mcp.pipedream.net/{external_user_id}/{app}` — templated per-user
+  URL + OAuth client-credentials); Composio (`connect.composio.dev/mcp` —
+  aggregator; TAS already integrates Composio directly).
+- **Dead/nonexistent, confirmed:** `mcp.zoho.com`, `mcp.digitalocean.com`,
+  `mcp.1password.com`, `mcp.moderntreasury.com`, `mcp.typeform.com`
+  (redirects to homepage; the real endpoint `api.typeform.com/mcp` shipped in
+  batch 3), `mcp.freshworks.com` (400), `mcp.canny.io` (serves HTML, not
+  MCP), `mcp.personio.com` (403 WAF).
+
+## Queued but never probed (speculative URLs — cheap batch-4 checks)
+
+These `mcp.{vendor}.com`-pattern guesses were staged in the drivers but never
+executed; no outcome exists anywhere:
+
+BambooHR (`mcp.bamboohr.com/mcp`), Bill.com (`mcp.bill.com/mcp`), Clio
+(`mcp.clio.com/mcp`), HiBob (`mcp.hibob.com/mcp`), Juro (`mcp.juro.com/mcp`),
+Qualtrics (`mcp.qualtrics.com/mcp`), Segment (`mcp.segment.com/mcp`), Sigma
+Computing (`mcp.sigmacomputing.com/mcp`), SpotDraft
+(`mcp.spotdraft.com/mcp`).

scripts/mcp-probe/driver-deep.py

@@ -0,0 +1,68 @@
+import subprocess, concurrent.futures as cf, json
+
+checks = {
+"typeform-loc": ("HEADLOC","https://mcp.typeform.com/mcp"),
+"drata-loc": ("HEADLOC","https://mcp.drata.com/mcp"),
+"expensify-loc": ("HEADLOC","https://mcp.expensify.com/mcp"),
+"dovetail-wk": ("GET","https://mcp.dovetail.com/.well-known/oauth-protected-resource"),
+"dovetail-get": ("GETH","https://mcp.dovetail.com/mcp"),
+"freshworks-wk": ("GET","https://mcp.freshworks.com/.well-known/oauth-protected-resource"),
+"docusign-wk": ("GET","https://mcp.docusign.com/.well-known/oauth-protected-resource"),
+"intuit-wk": ("GET","https://mcp.intuit.com/.well-known/oauth-protected-resource"),
+"personio-wk": ("GET","https://mcp.personio.com/.well-known/oauth-protected-resource"),
+"smartsheet-wk": ("GET","https://mcp.smartsheet.com/.well-known/oauth-protected-resource"),
+"lucid-wk": ("GET","https://mcp.lucid.app/.well-known/oauth-protected-resource"),
+"plaid-wk": ("GET","https://api.dashboard.plaid.com/.well-known/oauth-protected-resource"),
+"contentful-wk": ("GET","https://mcp.contentful.com/.well-known/oauth-protected-resource"),
+"pipedrive-wk": ("GET","https://mcp.pipedrive.com/.well-known/oauth-protected-resource"),
+"gorgias-wk": ("GET","https://mcp.gorgias.com/.well-known/oauth-protected-resource"),
+"pandadoc-wk": ("GET","https://mcp.pandadoc.com/.well-known/oauth-protected-resource"),
+"mercury-wk": ("GET","https://mcp.mercury.com/.well-known/oauth-protected-resource"),
+"jotform-wk": ("GET","https://mcp.jotform.com/.well-known/oauth-protected-resource"),
+"shortcut-wk": ("GET","https://mcp.shortcut.com/.well-known/oauth-protected-resource"),
+"serpapi-wk": ("GET","https://mcp.serpapi.com/.well-known/oauth-protected-resource"),
+"hunter-wk": ("GET","https://mcp.hunter.io/.well-known/oauth-protected-resource"),
+"airbyte-wk": ("GET","https://mcp.airbyte.ai/.well-known/oauth-protected-resource"),
+"heroku-wk": ("GET","https://mcp.heroku.com/.well-known/oauth-protected-resource"),
+"invideo-mcp": ("INIT","https://mcp.invideo.io/mcp"),
+"dialpad-alt": ("INIT","https://dialpad.com/mcp"),
+"digitalocean-alt": ("INIT","https://mcp.digitalocean.com/"),
+"moderntreasury-alt": ("INIT","https://app.moderntreasury.com/mcp"),
+"helpscout-alt": ("INIT","https://mcp.helpscout.net/mcp"),
+"zoho-alt": ("INIT","https://mcp.zohoapis.com/mcp"),
+"teamwork-wk": ("GET","https://mcp.ai.teamwork.com/.well-known/oauth-protected-resource"),
+"canny-wk": ("GET","https://mcp.canny.io/.well-known/oauth-protected-resource"),
+"courier-wk": ("GET","https://mcp.courier.com/.well-known/oauth-protected-resource"),
+"mux-wk": ("GET","https://mcp.mux.com/.well-known/oauth-protected-resource"),
+"telnyx-wk": ("GET","https://api.telnyx.com/.well-known/oauth-protected-resource"),
+"exa-wk": ("GET","https://mcp.exa.ai/.well-known/oauth-protected-resource"),
+"chilipiper-wk": ("GET","https://fire.chilipiper.com/api/fire-edge/v1/org/mcp/.well-known/oauth-protected-resource"),
+"smartsheet-init2": ("INIT","https://mcp.smartsheet.com/mcp"),
+"workato-get": ("GETH","https://mcp.workato.com/"),
+}
+init = json.dumps({"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"probe","version":"1.0"}}})
+
+def run(name, mode, url):
+    try:
+        if mode=="HEADLOC":
+            p=subprocess.run(["curl","-sk","-o","/dev/null","-D","-","--max-time","10","-X","POST","-H","Content-Type: application/json","--data",init,url],capture_output=True,text=True,timeout=15)
+            loc=[l for l in p.stdout.splitlines() if l.lower().startswith(("location","http"))]
+            return name," | ".join(loc)[:200]
+        if mode=="GET":
+            p=subprocess.run(["curl","-sk","--max-time","10",url],capture_output=True,text=True,timeout=15)
+            return name,p.stdout[:250].replace("\n"," ")
+        if mode=="GETH":
+            p=subprocess.run(["curl","-sk","-o","/dev/null","-D","-","--max-time","10",url,"-H","Accept: text/event-stream"],capture_output=True,text=True,timeout=15)
+            lines=p.stdout.splitlines()
+            return name,(lines[0] if lines else "")+" "+" ".join(l for l in lines if "auth" in l.lower())[:150]
+        if mode=="INIT":
+            p=subprocess.run(["curl","-sk","-o","/dev/null","-D","-","--max-time","10","-X","POST","-H","Content-Type: application/json","-H","Accept: application/json, text/event-stream","--data",init,url],capture_output=True,text=True,timeout=15)
+            lines=p.stdout.splitlines()
+            st=lines[0] if lines else "NOCONN"
+            www=next((l for l in lines if l.lower().startswith("www-authenticate")),"")
+            return name,st+" || "+www[:150]
+    except Exception as e:
+        return name,"ERR "+str(e)[:50]
+with cf.ThreadPoolExecutor(20) as ex:
+    for n,r in ex.map(lambda kv: run(kv[0],*kv[1]), checks.items()):
+        print(n,"\t",r)

scripts/mcp-probe/driver-shallow.py

@@ -0,0 +1,151 @@
+import json, subprocess, concurrent.futures as cf
+
+urls = {
+"figma": "https://mcp.figma.com/mcp",
+"gitlab": "https://gitlab.com/api/v4/mcp",
+"egnyte": "https://mcp-server.egnyte.com/mcp",
+"honeycomb": "https://mcp.honeycomb.io/mcp",
+"jotform": "https://mcp.jotform.com/",
+"lucid": "https://mcp.lucid.app/mcp",
+"lusha": "https://mcp.lusha.com/mcp",
+"make": "https://mcp.make.com",
+"microsoft-learn": "https://learn.microsoft.com/api/mcp",
+"mux": "https://mcp.mux.com",
+"newrelic": "https://mcp.newrelic.com/mcp",
+"pandadoc": "https://mcp.pandadoc.com/v1/mcp",
+"prisma": "https://mcp.prisma.io/sse",
+"sanity": "https://mcp.sanity.io",
+"statsig": "https://api.statsig.com/v1/mcp",
+"todoist": "https://ai.todoist.net/mcp",
+"wix": "https://mcp.wix.com/sse",
+"smartbear-bugsnag": "https://bugsnag.mcp.smartbear.com/mcp",
+"chilipiper": "https://fire.chilipiper.com/api/fire-edge/v1/org/mcp",
+"apify": "https://mcp.apify.com/",
+"airbyte": "https://mcp.airbyte.ai/mcp",
+"exa": "https://mcp.exa.ai/mcp",
+"clerk": "https://mcp.clerk.com/mcp",
+"cloudinary": "https://asset-management.mcp.cloudinary.com/mcp",
+"composio": "https://connect.composio.dev/mcp",
+"courier": "https://mcp.courier.com",
+"grafana": "https://mcp.grafana.com/mcp",
+"huggingface": "https://huggingface.co/mcp",
+"serpapi": "https://mcp.serpapi.com/mcp",
+"supabase": "https://mcp.supabase.com/mcp",
+"miro": "https://mcp.miro.com/",
+"teamwork": "https://mcp.ai.teamwork.com",
+"thoughtspot": "https://agent.thoughtspot.app/mcp",
+"lovable": "https://mcp.lovable.dev",
+"zapier": "https://mcp.zapier.com/api/mcp/mcp",
+"plaid": "https://api.dashboard.plaid.com/mcp/sse",
+"stytch": "https://mcp.stytch.dev/mcp",
+"telnyx": "https://api.telnyx.com/v2/mcp",
+"dialpad": "https://mcp.dialpad.com/mcp",
+"hunter": "https://mcp.hunter.io/mcp",
+"jam": "https://mcp.jam.dev/mcp",
+"netlify": "https://netlify-mcp.netlify.app/mcp",
+"invideo": "https://mcp.invideo.io/sse",
+"heroku": "https://mcp.heroku.com/mcp",
+"render": "https://mcp.render.com/mcp",
+"digitalocean": "https://mcp.digitalocean.com/mcp",
+"aws-knowledge": "https://knowledge-mcp.global.api.aws",
+"postman": "https://mcp.postman.com/minimal",
+"incident.io": "https://mcp.incident.io/mcp",
+"rootly": "https://mcp.rootly.com/sse",
+"buildkite": "https://mcp.buildkite.com/mcp",
+"semrush": "https://mcp.semrush.com/v1/mcp",
+"mailchimp": "https://mcp.mailchimp.com/",
+"canny": "https://mcp.canny.io/mcp",
+"helpscout": "https://mcp.helpscout.com/mcp",
+"knock": "https://mcp.knock.app/mcp",
+"onesignal": "https://mcp.onesignal.com/mcp",
+"zoho": "https://mcp.zoho.com/mcp",
+"smartsheet": "https://mcp.smartsheet.com/mcp",
+"shortcut": "https://mcp.shortcut.com/mcp",
+"moderntreasury": "https://mcp.moderntreasury.com/mcp",
+"intercom-x": "https://mcp.intuit.com/mcp",
+"deepwiki": "https://mcp.deepwiki.com/mcp",
+"context7": "https://mcp.context7.com/mcp",
+"tavily": "https://mcp.tavily.com/mcp/",
+"firecrawl": "https://mcp.firecrawl.dev/v2/mcp",
+"brightdata": "https://mcp.brightdata.com/mcp",
+"pipedream": "https://remote.mcp.pipedream.net",
+"vanta": "https://mcp.vanta.com/mcp",
+"atlan": "https://mcp.atlan.com/mcp",
+"workato": "https://mcp.workato.com/mcp",
+"fellow": "https://mcp.fellow.app/mcp",
+"productboard": "https://mcp.productboard.com/mcp",
+"gitbook": "https://mcp.gitbook.com/mcp",
+"zendesk-": "https://mcp.zendesk.com/mcp",
+"salesforce-": "https://mcp.salesforce.com/mcp",
+"calendly-": "https://mcp.calendly.com/mcp",
+"docusign-": "https://mcp.docusign.com/mcp",
+"typeform-": "https://mcp.typeform.com/mcp",
+"chargebee-": "https://mcp.chargebee.com/mcp",
+"ramp-": "https://mcp.ramp.com/mcp",
+"brex-": "https://mcp.brex.com/mcp",
+"deel-": "https://mcp.deel.com/mcp",
+"gusto-": "https://mcp.gusto.com/mcp",
+"greenhouse-": "https://mcp.greenhouse.io/mcp",
+"ashby-": "https://mcp.ashbyhq.com/mcp",
+"pipedrive-": "https://mcp.pipedrive.com/mcp",
+"zoominfo-": "https://mcp.zoominfo.com/mcp",
+"freshworks-": "https://mcp.freshworks.com/mcp",
+"front-": "https://mcp.frontapp.com/mcp",
+"gorgias-": "https://mcp.gorgias.com/mcp",
+"loops-": "https://mcp.loops.so/mcp",
+"launchdarkly-": "https://mcp.launchdarkly.com/mcp",
+"algolia-": "https://mcp.algolia.com/mcp",
+"snowflake-": "https://mcp.snowflake.com/mcp",
+"fivetran-": "https://mcp.fivetran.com/mcp",
+"dbt-": "https://mcp.getdbt.com/mcp",
+"sigma-": "https://mcp.sigmacomputing.com/mcp",
+"contentful-": "https://mcp.contentful.com/mcp",
+"iterable-": "https://mcp.iterable.com/mcp",
+"braze-": "https://mcp.braze.com/mcp",
+"customerio-": "https://mcp.customer.io/mcp",
+"segment-": "https://mcp.segment.com/mcp",
+"okta-": "https://mcp.okta.com/mcp",
+"1password-": "https://mcp.1password.com/mcp",
+"drata-": "https://mcp.drata.com/mcp",
+"carta-": "https://mcp.carta.com/mcp",
+"mercury-": "https://mcp.mercury.com/mcp",
+"bill-": "https://mcp.bill.com/mcp",
+"navan-": "https://mcp.navan.com/mcp",
+"expensify-": "https://mcp.expensify.com/mcp",
+"qualtrics-": "https://mcp.qualtrics.com/mcp",
+"surveymonkey-": "https://mcp.surveymonkey.com/mcp",
+"aha-": "https://mcp.aha.io/mcp",
+"dovetail-": "https://mcp.dovetail.com/mcp",
+"gainsight-": "https://mcp.gainsight.com/mcp",
+"salesloft-": "https://mcp.salesloft.com/mcp",
+"outreach-": "https://mcp.outreach.io/mcp",
+"docebo-": "https://mcp.docebo.com/mcp",
+"personio-": "https://mcp.personio.com/mcp",
+"hibob-": "https://mcp.hibob.com/mcp",
+"bamboohr-": "https://mcp.bamboohr.com/mcp",
+"rippling-": "https://mcp.rippling.com/mcp",
+"lattice-": "https://mcp.lattice.com/mcp",
+"clio-": "https://mcp.clio.com/mcp",
+"ironclad-": "https://mcp.ironcladapp.com/mcp",
+"spotdraft-": "https://mcp.spotdraft.com/mcp",
+"juro-": "https://mcp.juro.com/mcp",
+}
+
+init = json.dumps({"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"probe","version":"1.0"}}})
+
+def probe(name, url):
+    try:
+        p = subprocess.run(["curl","-sk","-o","/dev/null","-D","-","--max-time","12",
+            "-X","POST","-H","Content-Type: application/json","-H","Accept: application/json, text/event-stream",
+            "--data",init,url], capture_output=True, text=True, timeout=20)
+        head = p.stdout
+        lines = head.splitlines()
+        status = lines[0] if lines else "NOCONN"
+        www = next((l for l in lines if l.lower().startswith("www-authenticate")), "")
+        return name, url, status.strip(), www.strip()[:160]
+    except Exception as e:
+        return name, url, "ERR "+str(e)[:40], ""
+
+with cf.ThreadPoolExecutor(30) as ex:
+    for name,url,status,www in ex.map(lambda kv: probe(*kv), urls.items()):
+        print(f"{name}\t{status}\t{www}\t{url}")

scripts/mcp-probe/fetch-art.mjs

@@ -0,0 +1,61 @@
+import { readFileSync, writeFileSync, mkdirSync } from "node:fs";
+import { createHash } from "node:crypto";
+
+mkdirSync("art", { recursive: true });
+const rows = readFileSync("domains.txt", "utf8").trim().split("\n").map((l) => l.split(" "));
+
+const s2def = createHash("md5")
+  .update(new Uint8Array(await (await fetch("https://www.google.com/s2/favicons?domain=zzznotarealdomainxyz123.com&sz=128")).arrayBuffer()))
+  .digest("hex");
+
+const isPng = (b) => b.length > 8 && b[0] === 0x89 && b[1] === 0x50;
+const isIco = (b) => b.length > 4 && b[0] === 0 && b[1] === 0 && b[2] === 1;
+const isSvg = (b) => b.slice(0, 300).toString("latin1").toLowerCase().includes("<svg");
+const isJpg = (b) => b.length > 3 && b[0] === 0xff && b[1] === 0xd8;
+
+async function get(url) {
+  try {
+    const r = await fetch(url, {
+      redirect: "follow",
+      signal: AbortSignal.timeout(15000),
+      headers: { "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" },
+    });
+    if (!r.ok) return null;
+    return Buffer.from(await r.arrayBuffer());
+  } catch { return null; }
+}
+
+const results = [];
+const queue = [...rows];
+async function worker() {
+  for (;;) {
+    const row = queue.shift();
+    if (!row) return;
+    const [slug, domain] = row;
+    let src = null, buf = null, ext = "png";
+    for (const p of ["apple-touch-icon.png", "apple-touch-icon-precomposed.png"]) {
+      const b = await get(`https://${domain}/${p}`);
+      if (b && isPng(b) && b.length >= 1000) { src = "apple-touch-icon"; buf = b; break; }
+    }
+    if (!buf) {
+      const b = await get(`https://www.google.com/s2/favicons?domain=${domain}&sz=128`);
+      if (b && b.length >= 500) {
+        const h = createHash("md5").update(new Uint8Array(b)).digest("hex");
+        if (h !== s2def && (isPng(b) || isIco(b) || isJpg(b))) {
+          src = "s2"; buf = b;
+          ext = isPng(b) ? "png" : isJpg(b) ? "jpg" : "ico";
+        }
+      }
+    }
+    if (buf) {
+      writeFileSync(`art/${slug}.${ext}`, buf);
+      results.push([slug, src, buf.length, ext]);
+    } else results.push([slug, "FAIL", 0, ""]);
+  }
+}
+await Promise.all(Array.from({ length: 10 }, worker));
+results.sort((a, b) => a[0].localeCompare(b[0]));
+for (const r of results) console.log(r.join("\t"));
+const c = (k) => results.filter((r) => r[1] === k).length;
+console.log(`-- apple-touch-icon: ${c("apple-touch-icon")}, s2: ${c("s2")}, FAIL: ${c("FAIL")}`);
+writeFileSync("art-results.json", JSON.stringify(results));