Mask non-native log-pipeline NDJSON envelopes (.dc4/.dc5)

[zelmario]

Problem

Some logging pipelines (e.g. a Kubernetes log collector) don't emit native mongod structured JSON. They re-wrap each event in a custom NDJSON envelope — mongo_cluster / replicaset / pod identity plus the original event split into message / query / query_meta — and pretty-print each record across many lines.

Because that isn't a native mongod JSON line, the line-by-line discovery in _discover_log_file never parses it (each line fails json.loads, falling back to the email/IP freetext scan only). The result: the real db/collection/host, the embedded query command, the CRUD document images, and schema field names pass straight through to the obfuscated output.

Real-world trigger: an FTDC support ticket whose top-level .dc4/.dc5 files (these envelopes, not native logs) leaked the namespace, the slow-query command, projection field names, and inserted-document images.

Fix

Detect the envelope and route it to dedicated discover/replace passes:

• _looks_like_envelope — cheap head sniff (mongo_cluster + pod/replicaset + a payload key).
• _iter_envelope_records — streams the concatenated (pretty-printed) objects via raw_decode, resyncing past a malformed record.

The query command and the reconstructed attr={...} document image are never safe to keep field-by-field — one missed key leaks — so they are redacted wholesale. Only provably non-sensitive content is kept:

Field	Treatment
mongo_cluster / replicaset / pod	masked identifiers
query_meta.ns	masked database.collection
numeric perf counters (durationMillis, keysExamined, docsExamined, nreturned, reslen, locks, …)	kept verbatim
planSummary	plan STAGE kept, index key spec redacted → IXSCAN { <redacted> } (in query_meta and the message text)
query	<redacted-query>
message attr={...}	attr=<redacted> (human log text kept)
every other query_meta string (appName, comment, …)	<redacted>

Output is re-emitted as NDJSON (one record per line — now re-parseable). Masked identifiers round-trip through cluster_mapping.json; the <redacted*> markers intentionally have no mapping.

Scope / safety

Purely additive (209 insertions, 0 deletions): native mongod logs and FTDC are untouched — the two hooks in _discover_log_file / _replace_log_file only divert when _looks_like_envelope matches. Independent of #1 and #2.

So the analysis still sees the shape of the problem (collection X, COLLSCAN, examined 1.2M → returned 1, 250ms) but never the customer's fields, values, or documents.