brotli-compress .socket.facts.json on upload (#1291)

* refactor: brotli-compress .socket.facts.json on upload

Compress `.socket.facts.json` with brotli at upload time, just before
`fetchCreateOrgFullScan` POSTs the multipart `/full-scans` body. Coana
keeps writing plain JSON; the local readers
(`extractTier1ReachabilityScanId`, `extractReachabilityErrors`) keep
reading plain JSON; only the on-the-wire bytes between socket-cli and
depscan change. depscan transparently decompresses at the api-v0
multipart ingest boundary in a coordinated change.

Why:
* Mono-repo `.socket.facts.json` files routinely exceed 60MB. On a
  representative simple-npm fixture, brotli compresses 150,748 bytes
  to 19,971 (~86.8% reduction); production mono-repos see a much
  larger absolute saving.
* Coana has a second consumer downstream - teaching coana to write
  `.br` would break it. Brotli on the wire-only path keeps coana's
  contract invariant.
* `tier1ReachabilityScanId` and reachability-error reporting still
  read plain JSON locally; no brotli round-trip on those paths.
* Compression is a transport detail owned by the upload site; cleanup
  is one `finally` block.

Implementation:
* `src/utils/coana.mts` - new exported
  `compressSocketFactsForUpload(scanPaths)` returning `{ paths, cleanup }`.
  Per `.socket.facts.json` entry, `mkdtempSync` a fresh `.socket-br-XXX/`
  directory inside the source's parent dir (NOT under `os.tmpdir()` -
  see below), `brotliCompressSync` bytes to
  `${td}/.socket.facts.json.br`, swap the path. Other paths pass through.
  Missing facts paths pass through. Cleanup is idempotent with
  `force: true`.
* `src/commands/scan/handle-create-new-scan.mts` - wraps the
  `fetchCreateOrgFullScan` call in
  `try { compress; upload; } finally { cleanup(); }`. Cleanup runs on
  success, throw, or any abort path.

Why the temp lives next to the source:

The SDK computes `path.relative(cwd, brPath)` for the multipart name
field. depscan's multipart ingest (`addStreamEntry`) checks
`containsTraversal(unixified)` and bails on any `..` segment. A
tmpdir-based path resolves to `../../../var/folders/...`, gets dropped
to `unmatchedFiles`, and the SocketFacts content never lands in the
scan. Putting the temp inside `path.dirname(originalFactsPath)`
produces a relative path like `.socket-br-XXX/.socket.facts.json.br` -
traversal-free, ingests cleanly.

Tests:
* `src/utils/coana.test.mts` - 16 cases.
  - `compressSocketFactsForUpload` x 5: round-trip JSON via
    `brotliDecompressSync`, basename swap to `.socket.facts.json.br`,
    non-facts paths pass through, missing facts paths pass through,
    cleanup idempotent, mixed-entry order. Pins the contract that the
    temp lives in a subdir of the source's parent (traversal-free).
  - `extractTier1ReachabilityScanId` x 7: plain JSON, missing file,
    missing field, null, empty/blank, trim, numeric coercion.
  - `extractReachabilityErrors` x 4: extraction, missing file,
    no-components, no-inner-reachability.

This change requires the matching depscan multipart decompression
patch on the receiving side; that change ships first.

* fix: write brotli .br as sibling of source, not in temp subdir

Previous form wrapped each `.br` in a per-scan `mkdtempSync` subdir
under the source dir for concurrency isolation. That created a
directory-handling asymmetry on depscan's side: a wire path of
`dirA/.socket.facts.json.br` flattened to `.socket.facts.json` at
root via depscan's basename-strip, while plain `dirA/.socket.facts.json`
preserved the dir.

depscan dropped the basename-strip in the corresponding PR; switch
to writing `.br` as a sibling of the source so wire and storage
paths match for both branches. Net effect: a brotli upload from
`<source>/.socket.facts.json` lands at `<source>/.socket.facts.json`
in the manifest tarball - identical to plain.

Concurrent-scan note: coana already writes to a single source path,
so the source `.socket.facts.json` itself is racy under concurrent
runs against the same dir; the sibling `.br` doesn't introduce a
new race that wasn't already there.

* `src/utils/coana.mts`: `compressSocketFactsForUpload` writes
  `${p}.br` next to the source; `cleanup()` does
  `rm(brPath, { force: true })` per file. Drops `mkdtemp` import
  that's no longer used.
* `src/utils/coana.test.mts`: directory-shape assertion replaced
  with `swappedPath === ${input}.br` (sibling). First test now
  also asserts the source survives `cleanup()`. 16 cases.

* fix: clean up partial brotli siblings on compress failure

Per Cursor Bugbot finding on src/utils/coana.mts: if any brotli
pipeline in the `Promise.all` batch rejects, the helper rejected
before returning its `cleanup` callback, so already-completed
sibling `.br` files were orphaned next to their source.

Three changes that together make the helper failure-safe:

* Track the intended `.br` path BEFORE the pipeline starts. Previous
  code pushed to `brPaths` after `await pipeline(...)`, so a sibling
  that the pipeline started writing but didn't finish was invisible
  to cleanup. `rm({ force: true })` no-ops on missing paths, so
  tracking the intent is safe.
* Switch the parallel batch from `Promise.all` to `Promise.allSettled`.
  With `all`, the first rejection bubbled out while sibling pipelines
  were still writing bytes; calling `cleanup()` then would race the
  live writes and could `rm` a sibling only to have it re-created
  immediately. `allSettled` waits for every pipeline to settle, so
  cleanup sees a stable set of files to remove.
* On batch failure, run `cleanup()` internally before re-throwing so
  the caller (which only gets a chance to call cleanup when the
  helper resolves) doesn't have to. Add `recursive: true` to the rm
  call so a defensively-occupied directory at a `.br` path doesn't
  halt cleanup partway through the list.

Test added: `removes partial .br siblings if compression fails
mid-batch` — pre-occupies entry B's `.br` destination with a
directory so its `createWriteStream` rejects with EISDIR, then
asserts that entry A's `.br` is not orphaned on disk after the
batch rejects and both source files survive untouched.

* chore(release): 1.1.98

Bump version to 1.1.98 and add changelog entry for the brotli
upload-compression refactor in this PR.

Author: barslev

CHANGELOG.md

@@ -10,6 +10,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - **`socket manifest bazel [beta]`** — Generate Bazel JVM SBOM manifests by running `bazel query` against discovered Maven repos in a Bazel workspace. Closes the inline-Maven-declaration gap that lockfile-only parsing misses for repos like envoy, ray, tensorflow, tink-java, and or-tools. Auto-detects Bzlmod and legacy `WORKSPACE`.
 - **`socket scan create --auto-manifest`** now covers Bazel workspaces in addition to Gradle/Scala/Kotlin/Conda. Repos with `MODULE.bazel`, `WORKSPACE`, or `WORKSPACE.bazel` are detected automatically and their Maven dependencies extracted as part of the standard scan-create flow.
 
+## [1.1.98](https://github.com/SocketDev/socket-cli/releases/tag/v1.1.98) - 2026-05-20
+
+### Changed
+- `socket scan create --reach` now uploads the reachability facts file as brotli on the wire, shrinking mono-repo upload sizes by roughly 85% with no change to the on-disk or stored format. Faster scan submissions on slow connections.
+
 ## [1.1.97](https://github.com/SocketDev/socket-cli/releases/tag/v1.1.97) - 2026-05-18
 
 ### Changed

package.json

@@ -1,6 +1,6 @@
 {
   "name": "socket",
-  "version": "1.1.97",
+  "version": "1.1.98",
   "description": "CLI for Socket.dev",
   "homepage": "https://github.com/SocketDev/socket-cli",
   "license": "MIT AND OFL-1.1",

src/commands/scan/handle-create-new-scan.mts

@@ -15,6 +15,7 @@ import { outputCreateNewScan } from './output-create-new-scan.mts'
 import { performReachabilityAnalysis } from './perform-reachability-analysis.mts'
 import constants from '../../constants.mts'
 import { checkCommandInput } from '../../utils/check-input.mts'
+import { compressSocketFactsForUpload } from '../../utils/coana.mts'
 import { findSocketYmlSync } from '../../utils/config.mts'
 import { getPackageFilesForScan } from '../../utils/path-resolve.mts'
 import { readOrDefaultSocketJson } from '../../utils/socket-json.mts'
@@ -279,28 +280,40 @@ export async function handleCreateNewScan({
     tier1ReachabilityScanId = reachResult.data?.tier1ReachabilityScanId
   }
 
-  const fullScanCResult = await fetchCreateOrgFullScan(
-    scanPaths,
-    orgSlug,
-    {
-      commitHash,
-      commitMessage,
-      committers,
-      pullRequest,
-      repoName,
-      branchName,
-      scanType: reach.runReachabilityAnalysis
-        ? constants.SCAN_TYPE_SOCKET_TIER1
-        : constants.SCAN_TYPE_SOCKET,
-      workspace,
-    },
-    {
-      cwd,
-      defaultBranch,
-      pendingHead,
-      tmp,
-    },
-  )
+  // Brotli-compress any .socket.facts.json paths in scanPaths just before
+  // upload. depscan's api-v0 multipart boundary streams brotli decode based
+  // on the .br filename suffix. Coana keeps writing plain .socket.facts.json
+  // on disk, so the local read paths (extractTier1ReachabilityScanId,
+  // extractReachabilityErrors) stay correct. The cleanup() in the finally
+  // block removes the temp dirs whether the upload succeeded or threw.
+  const compressed = await compressSocketFactsForUpload(scanPaths)
+  let fullScanCResult: Awaited<ReturnType<typeof fetchCreateOrgFullScan>>
+  try {
+    fullScanCResult = await fetchCreateOrgFullScan(
+      compressed.paths,
+      orgSlug,
+      {
+        commitHash,
+        commitMessage,
+        committers,
+        pullRequest,
+        repoName,
+        branchName,
+        scanType: reach.runReachabilityAnalysis
+          ? constants.SCAN_TYPE_SOCKET_TIER1
+          : constants.SCAN_TYPE_SOCKET,
+        workspace,
+      },
+      {
+        cwd,
+        defaultBranch,
+        pendingHead,
+        tmp,
+      },
+    )
+  } finally {
+    await compressed.cleanup()
+  }
 
   const scanId = fullScanCResult.ok ? fullScanCResult.data?.id : undefined
 

src/utils/coana.mts

@@ -3,6 +3,11 @@
  * Manages reachability analysis via Coana tech CLI.
  *
  * Key Functions:
+ * - compressSocketFactsForUpload: Brotli-compress any .socket.facts.json
+ *   entries in scanPaths just before upload, returning swapped paths plus a
+ *   cleanup callback. Coana keeps writing plain JSON; the on-the-wire form
+ *   to depscan is brotli (api-v0 decodes at the multipart boundary).
+ * - extractReachabilityErrors: Extract per-component reachability errors
  * - extractTier1ReachabilityScanId: Extract scan ID from socket facts file
  *
  * Integration:
@@ -11,8 +16,105 @@
  * - Extracts tier 1 reachability scan identifiers
  */
 
+import { createReadStream, createWriteStream, existsSync } from 'node:fs'
+import { rm } from 'node:fs/promises'
+import path from 'node:path'
+import { pipeline } from 'node:stream/promises'
+import { createBrotliCompress } from 'node:zlib'
+
 import { readJsonSync } from '@socketsecurity/registry/lib/fs'
 
+import constants from '../constants.mts'
+
+const { DOT_SOCKET_DOT_FACTS_JSON } = constants
+
+export type CompressedScanPaths = {
+  paths: string[]
+  cleanup: () => Promise<void>
+}
+
+/**
+ * For each `.socket.facts.json` in `scanPaths`, stream-brotli-compress a
+ * sibling `.socket.facts.json.br` next to the original file and swap its
+ * path in. Other paths pass through unchanged. Missing files also pass
+ * through unchanged (the upload will fail downstream with the same error
+ * it would have).
+ *
+ * Streaming + worker-thread compression keeps the event loop responsive:
+ * default brotli quality (11) on a 60+MB facts file takes multiple seconds
+ * of CPU, which would otherwise freeze the spinner / signal handlers /
+ * any concurrent work.
+ *
+ * The `.br` lives next to the source rather than under the OS temp dir
+ * because depscan's multipart ingest (`addStreamEntry`) rejects entries
+ * whose names contain `..` traversal segments. The SDK computes the
+ * multipart entry name via `path.relative(cwd, brPath)`, so an OS-tmpdir
+ * temp path turns into `../../../var/folders/...` and gets dropped as
+ * `unmatchedFiles`. Sibling-write keeps the relative path inside cwd, and
+ * keeps the directory shape symmetric with the plain `.socket.facts.json`
+ * upload (depscan strips only the `.br` suffix at ingest, so
+ * `<dir>/.socket.facts.json.br` and `<dir>/.socket.facts.json` resolve to
+ * the same storage path).
+ *
+ * Concurrent scans against the same source directory are already racy on
+ * `.socket.facts.json` itself (coana writes to a single path), so the
+ * sibling `.br` doesn't introduce a new race.
+ *
+ * Caller MUST `await cleanup()` (typically in a `finally` block) once the
+ * upload completes — successful or not — to remove the sibling files.
+ */
+export async function compressSocketFactsForUpload(
+  scanPaths: string[],
+): Promise<CompressedScanPaths> {
+  const brPaths: string[] = []
+  const cleanup = async () => {
+    const targets = brPaths.splice(0)
+    // `recursive: true` defends against the (defensive) case where a sibling
+    // path was somehow created as a directory — `rm` would otherwise throw
+    // on the first such entry and skip the rest. `force: true` no-ops on
+    // missing paths so the function stays idempotent.
+    await Promise.all(targets.map(t => rm(t, { recursive: true, force: true })))
+  }
+  // Use `allSettled` (not `all`) so a failure in one entry doesn't leak the
+  // others' in-flight pipelines past our `catch`. If we used `all`, the
+  // first rejection would bubble out while sibling pipelines were still
+  // writing bytes — `cleanup()` would race with those writes and could
+  // remove a `.br` only to have it re-created after we returned.
+  const results = await Promise.allSettled(
+    scanPaths.map(async p => {
+      if (path.basename(p) !== DOT_SOCKET_DOT_FACTS_JSON) {
+        return p
+      }
+      if (!existsSync(p)) {
+        return p
+      }
+      const brPath = `${p}.br`
+      // Track the sibling path BEFORE the pipeline starts so a
+      // partially-written `.br` is removed even if the pipeline rejects.
+      // `rm({ force: true })` no-ops on missing files, so tracking before
+      // creation is safe.
+      brPaths.push(brPath)
+      await pipeline(
+        createReadStream(p),
+        createBrotliCompress(),
+        createWriteStream(brPath),
+      )
+      return brPath
+    }),
+  )
+  const failure = results.find(
+    (r): r is PromiseRejectedResult => r.status === 'rejected',
+  )
+  if (failure) {
+    // All pipelines have settled, so cleanup() can safely remove every
+    // `.br` we tracked (succeeded or partial) without racing live writes.
+    await cleanup()
+    throw failure.reason
+  }
+  const paths = results.map(r => (r as PromiseFulfilledResult<string>).value)
+  return { paths, cleanup }
+}
+
 export type ReachabilityError = {
   componentName: string
   componentVersion: string