HTTP usage metering: attribution is by project name — harden docs/e2e (#287) + investigate malformed subjects

[ecv]

Corrected understanding

Important

An earlier version of this issue claimed gateway usage metering emitted unresolvable subjects and that "all usage is rejected." That was wrong. Confirmed against the consumer code (milo-os/billing, internal/controller/consumer).

• The consumer attributes usage by project name: consumer.go strips projects/ from the CloudEvent subject and matches the remainder against BillingAccountBinding.spec.projectRef.name (billingaccountbinding_cache.go).
• The NSO VRL already builds the subject as projects/<project-name> — correct, no change needed.
• attribution_failure fires when a project has no active BillingAccountBinding (or its account isn't Ready). So the high reject volume on prod is dominated by unbilled / test projects — expected, not a derivation bug. Accepted usage is republished to billing.usage.<project>.valid (no success counter), which is why "no accepted metric" looked like total failure but isn't.
• The production broker outage (datum-cloud/infra#3448) is a separate problem — event volume against an unsized single-replica stream, independent of attribution.

What's actually actionable

1. Harden the model against silent regression — done in test(billing): pin usage subject to project name + fix metering doc #287: pins the billing e2e subject to projects/<project-name> (it previously accepted any subject) and corrects the design doc, which called project_name a mere "dimension."
2. Investigate malformed subjects. A subset of rejected project values are malformed — leading-underscore _datum-cloud, 0-datum-internal-anchor. These are a real (smaller) derivation/injection bug: is idx.ProjectNames[dsNS] returning a malformed name, or does a specific route class produce it?
3. Confirm no billable project is wrongly rejected. Cross-check the rejected project set against projects that do have an active BillingAccountBinding; any overlap is a real attribution bug beyond the malformed set.

[ecv]

Root cause pinpointed

The bug is in the VRL subject derivation, config/tools/billing-usage-collector/kustomization.yaml:

project_id = replace!(to_string(httproute_namespace), r'^ns-', "")   # correct — project UID from route_name
if project_name != "" {
  project_id = project_name                                          # BUG — promotes the display-name dimension to the attribution subject
}
subject = "projects/" + project_id

Per this repo's design doc (docs/enhancements/http-metering/http-traffic-metering.md, "Surfacing Signals from the Edge", point 3), project_name is meant to be a dimension only — the human-readable name the extension server stamps into route metadata. The billing subject must be the project UID carried in route_name (ns-<uid> → <uid>). The if project_name != "" override conflates the two, so the subject becomes projects/<display-name>, which the billing consumer can't resolve → 100% attribution_failure.

Confirmed against production: the rejected project label values are real display names (datum-cloud, tunneldemo-…, jreese-test-…), not UIDs.

Fix

1. Delete the override (the if project_name != "" { project_id = project_name } block). Subject reverts to projects/<uid>; project_name stays in the dimensions map only.

2. Harden the e2e so it would catch this. test/e2e/billing/chainsaw-test.yaml already asserts subject == projects/<namespace-uid> and requires project_name present — but if the fixture's project_name equals the namespace UID, the buggy override produces the same subject and the test stays green (this is why it regressed silently). The fixture must set project_name to a value distinct from the namespace UID so the assertion fails if the subject is ever built from the display name.

3. Sync the pinned mirror in infra (apps/billing-vector-agent/components/billing-usage-collector-logs/kustomization.yaml) once this lands + passes e2e.

Split out (not this bug)

• unknown_meter (~652k rejects): meter registration, not derivation. The Service/ServiceConfiguration are in Draft phase per the doc — the four networking.datumapis.com/gateway/* MeterDefinitions may not be published. Track separately (catalog side).
• Aggregation: the pipeline emits 4 CloudEvents per request with no aggregation; the doc only mitigates transport volume (on-node batching), not event count. Unbounded-per-request is by-design today and is what magnified this into a broker outage. Worth a dedicated enhancement.

[ecv]

Correction — supersedes my earlier "delete the project_name override" comment above.

That pinpoint was wrong. The consumer attributes by project name, so the VRL's project_name subject is correct — the override must stay. I'd read the contract from the NSO producer doc + a mock-only e2e instead of the consumer code. Issue re-scoped above; the durable fix is the e2e/doc hardening in #287, not a VRL change.