feat(srv6): cut over to a TC-BPF uFMT 48+16 uSID datapath - #273
Open
privateip wants to merge 6 commits into
Open
feat(srv6): cut over to a TC-BPF uFMT 48+16 uSID datapath#273privateip wants to merge 6 commits into
privateip wants to merge 6 commits into
Conversation
Galactic's per-endpoint seg6local ingress route model installs one static kernel route per (VPC, VPCAttachment) pair -- O(N) FIB entries per node, and no way to decode the shared Function+Argument uSID slot datum-cloud/enhancements#740 specifies (uFMT 48+16, RFC 9800 REPLACE-CSID). Vanilla seg6local can't parse a sub-field argument dynamically, so closing that gap needed a real in-kernel datapath, not a config change. Adds a new eBPF/TC-BPF ingress program (internal/plumbing/ebpf/) that matches a node's own uSID Block/Node-ID, reads Function/Argument directly from the unmutated packet, resolves the Argument to a Linux VRF via bpf_fib_lookup, and redirects into the pod's veth or tap interface -- O(1) FIB cost per node regardless of tenant count. ComputeSID (internal/plumbing/srv6/usid.go) now emits the real uFMT 48+16 layout via a new local per-node Argument allocator (allocateArgument, internal/cni/bgp.go) sourced from BGPVRFInstance CRD state rather than the eBPF maps themselves, so the encoder never depends on the datapath being loaded on a given node. galactic-cni registers/unregisters each attachment's vrf_table entry on ADD and rollback; a new GC sweep reclaims entries orphaned by deleted BGPVRFInstances. Also fixes two correctness bugs found deploying this to a live ContainerLab fabric -- the netlink watcher never re-verified an interface an external event (an FRR restart) had silently detached from, and RouteEgressAdd's full SEG6 encap mode pushed a Segment Routing Header that usid.c's fixed-width header strip didn't account for, misreading it as the inner IP version on every cross-region packet -- plus a bpf_fib_lookup neighbor-priming gap for pods whose address never otherwise triggered ARP/NDP. Separately, EVPN Type 5's GWIPAddress can only carry a gateway matching its own NLRI's address family, so an IPv4 prefix's SRv6 SID was silently dropped in transit; it's now carried via a new RFC 9252 Prefix-SID path attribute instead, independent of prefix family. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The native "Build" CI job runs `task build`, which shells out to bpf2go to compile internal/plumbing/ebpf/prog/usid.c and strip the result with llvm-strip. ubuntu-latest ships clang but not llvm-strip, so the job failed with "executable file not found in $PATH". The Docker-based image jobs already install both in their Dockerfile, which is why only the native Build job was affected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
registerEBPFDatapath (internal/cni/bgp.go) is now the only forwarding path and requires locator_table/function_table/vrf_table to already be pinned under attach.PinDir. In production that's guaranteed ahead of time by the CNI DaemonSet's long-running `/galactic-cni run` container (config/cni/daemonset.yaml), but TestCNITapInterface spins up its own bare pod and invokes CNI ADD directly, so nothing ever loaded/pinned those maps -- CNI ADD failed with "open pinned map \"vrf_table\": no such file or directory". Start the same control daemon inside the test pod first and wait for its maps to be pinned, mirroring what the DaemonSet does before any pod attach can happen for real. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The eBPF control daemon started in the previous commit failed with "mkdir /sys/fs/bpf/galactic: no such file or directory" -- a pod's own mount namespace can't create /sys/fs/bpf out of thin air (same reason config/cni/daemonset.yaml's bpf-fs hostPath volume comment gives: the mount has to already exist on the node). A real node's OS/kubelet setup mounts bpffs at boot; Kind node containers don't, so ci.sh now mounts it explicitly, and the test pod gets a bpf-fs hostPath volume (mirroring the DaemonSet's own) so it can see that mount. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kubectl run's --overrides merge replaces the whole generated "containers" list rather than merging into it once "containers" is set at all, so adding the bpf-fs volumeMount there silently dropped the container's image/command/privileged fields that had been set via the usual --image/--command/--privileged flags -- confirmed via `kubectl run ... --dry-run=client -o yaml`, which showed an empty container (name + volumeMounts only). `kubectl run failed: exit status 1` in CI was the API server rejecting that incomplete pod spec. Moving image/command/privileged into the same --overrides JSON fixes it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
allocateArgument's list-then-write BGPVRFInstance creation race let two concurrent CNI ADDs claim the same eBPF vrf_table Argument: the prior tie-break only let the lexicographically-losing name error out, so a tight enough create/check interleaving let both attachments pass and permanently share one VRFID, misdelivering one VPC's decapped traffic into the other's VRF. checkArgumentCollision now errors on any other same-VRFID instance regardless of name order, guaranteeing at least one side always detects it and retries via the normal failed-ADD path. That failed-ADD rollback had its own gap: retryK8sOps can re-run the whole publishBGPStateK8s closure without re-registering the eBPF entry, so a later attempt's collision failure could trigger cleanup to unregister a vrf_table slot the colliding (winning) attachment had since overwritten. unregisterEBPFDatapath now takes the caller's own, freshly-recomputed VRF table id and only deletes the entry if it still matches. SweepEBPFVRFTable folded "no BGPRouter found for this node" into the same case as "nothing is live," reconciling the whole vrf_table against an empty live set and blackholing every attachment on a transient router-listing hiccup. It now skips the sweep tick entirely when no router is found, rather than treating that as ground truth. Finally, BuildDesiredRouter aborted the entire DesiredRouter -- every peer, VRF, and advertisement -- when one BGPAdvertisement's VRFID fell outside the eBPF datapath's 12-bit Argument range (as any pre-cutover object allocated under the old, wider VRFID scheme would). That advertisement is now skipped with a warning instead of failing the whole router. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
seg6localuSID ingress route model (one static kernel route per(VPC, VPCAttachment), O(N) FIB entries per node) with a new eBPF/TC-BPF datapath (internal/plumbing/ebpf/) implementing theuFMT 48+16uSID carrier from feat: Datum Cloud IPv6 addressing plan enhancements#740 (RFC 9800 REPLACE-CSID / Option 2) — O(1) FIB cost per node regardless of tenant count.ComputeSID(internal/plumbing/srv6/usid.go) now emits the real uFMT 48+16 layout, backed by a new local per-node Argument allocator (allocateArgument,internal/cni/bgp.go) sourced fromBGPVRFInstanceCRD state.galactic-cniregisters/unregisters each attachment'svrf_tableentry on ADD/rollback; a new GC sweep (internal/gc) reclaims entries orphaned by deletedBGPVRFInstances.seg6localpath and its feature flags (GALACTIC_CNI_ENABLE_EBPF_DATAPATH,GALACTIC_CNI_EBPF_OBSERVE_ONLY) are deleted outright rather than kept behind a flag/observe-only mode.RouteEgressAdd's full SEG6 encap mode pushed a Segment Routing Header thatusid.c's fixed-width header strip didn't account for. Also adds abpf_fib_lookupneighbor-priming fix for pods whose address never otherwise triggered ARP/NDP.internal/runtime/gobgp), sinceGWIPAddresscan only carry a gateway matching its own NLRI's address family and was silently dropping the SID for IPv4 prefixes.vpcis a 48-bit identifier (it's 16-bit, cluster-scoped) and adds a package-leveldoc.gooverview forinternal/plumbing/ebpf/.🤖 Generated with Claude Code