diff --git a/README.md b/README.md index efb7278..41d573f 100644 --- a/README.md +++ b/README.md @@ -735,14 +735,16 @@ Pass the topology scheme and file on the CLI, or set the same values under l8k generate --user-config cluster-config.yaml \ --spectrum-x RA2.3 \ --topology-scheme 2-tier \ - --ip-version ipv4 \ + --ip-version ipv6 \ --topology-file ./topology.json ``` `profile.spectrumX.hostFirstOctet` is config-only. When omitted, l8k uses `172` -for 2-tier IPv4 allocation and `10` for 3-tier IPv4 allocation. `ipVersion: -ipv6` is accepted in the config and CLI, but Spectrum-X CIDRPool rendering -currently supports IPv4 static allocations only. +for 2-tier IPv4 allocation and `10` for 3-tier IPv4 allocation. It has no effect +on IPv6 allocation. IPv6 uses the standard `fd02:00PP:RRDD:SSHH::peer` layout, +with a `/64` per node, host candidate `::1`, leaf gateway `::2`, and a `/40` +CIDRPool per rail or rail-plane. Generated IPv6 routes use `/32` for a single +plane and `/24` for dual- or quad-plane deployments. CIDRPool allocation requires exact, case-sensitive equality between selected `clusterConfig.workerNodes` values and topology host endpoint `node` values. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index cd0fdf7..be3dc0b 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -71,7 +71,7 @@ Discovery also accepts the profile and Spectrum-X flags below. Explicit flags ov | `--multiplane-mode` | `none`, `swplb`, or `hwplb`. | | `--number-of-planes` | Plane count for Spectrum-X. | | `--topology-scheme` | `2-tier` or `3-tier` for topology-driven CIDRPool allocation. | -| `--ip-version` | `ipv4` or `ipv6`. CIDRPool rendering currently supports IPv4. | +| `--ip-version` | `ipv4` for per-node `/31` allocation or `ipv6` for per-node `/64` allocation. | | `--topology-file` | Path to spcx-gen/reference-generator or contract-compliant NVIDIA AIR topology JSON. The format is detected from its structure. | | `--spectrum-x-config` | Full ConfigMap YAML or raw `data.profile` YAML. Required for RA2.3. | | `--spectrum-x-configmap-name` | ConfigMap name when `--spectrum-x-config` is raw profile YAML. | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 194358a..4401559 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -267,7 +267,7 @@ profile: | `spectrumX.multiplaneMode` | `none`, `swplb`, or `hwplb`. | | `spectrumX.numberOfPlanes` | `1`, `2`, or `4`. | | `spectrumX.topologyType` | `2-tier` or `3-tier`. | -| `spectrumX.ipVersion` | `ipv4` or `ipv6`; CIDRPool rendering currently supports IPv4. | +| `spectrumX.ipVersion` | `ipv4` for per-node `/31` allocation or `ipv6` for per-node `/64` allocation. | | `spectrumX.hostFirstOctet` | Config-only first octet for generated IPv4 topology addressing. | | `spectrumX.topologyFile` | Path to spcx-gen/reference-generator or contract-compliant NVIDIA AIR topology JSON. The format is detected from its structure; relative paths resolve from the config file. | | `spectrumX.configMapName` / `profile` | RA2.3 ConfigMap name and embedded profile data. | diff --git a/docs/user/spectrum-x.md b/docs/user/spectrum-x.md index 6ebaccc..1acba18 100644 --- a/docs/user/spectrum-x.md +++ b/docs/user/spectrum-x.md @@ -166,11 +166,30 @@ l8k generate \ --spectrum-x RA2.3 \ --spectrum-x-config ./spectrum-x-profile-configmap.yaml \ --topology-scheme 2-tier \ - --ip-version ipv4 \ + --ip-version ipv6 \ --topology-file ./topology.json ``` -IPv4 CIDRPool rendering is supported. IPv6 is accepted in config for forward compatibility but is not rendered into CIDRPools yet. +Both `ipv4` and `ipv6` generate complete nv-ipam CIDRPools. IPv4 preserves the +existing per-node `/31` allocation. IPv6 uses the standard Spectrum-X layout: + +```text +fd02:00PP:RRDD:SSHH::peer/64 +``` + +`PP`, `RR`, `DD`, `SS`, and `HH` are the zero-based plane, rail, pod, SU, and +host indices encoded as one byte each. Two-tier topologies require `DD=00`. +The host candidate is `::1`, the connected leaf and gateway are `::2`, and the +static allocation prefix is the canonical `/64` network. Each pool covers a +rail or rail-plane with a `/40`. The route is `/32` for a single-plane +deployment and `/24` for a dual- or quad-plane deployment. + +In `swplb`, the plane is encoded and l8k emits one pool per rail-plane. In +`none` and `hwplb`, the address plane is zero and l8k emits one pool per rail. +`profile.spectrumX.hostFirstOctet` affects IPv4 only. The current topology +contract provides the fields required by this standard layout; alternative +platform-specific layouts that require additional topology fields are not +generated. ### Troubleshooting CIDRPool allocation errors diff --git a/pkg/cmd/schema.go b/pkg/cmd/schema.go index cf27dc1..3bbbd18 100644 --- a/pkg/cmd/schema.go +++ b/pkg/cmd/schema.go @@ -160,7 +160,7 @@ var schemaCmd = &cobra.Command{ "--ip-version": { Type: "string", Default: config.SpectrumXIPVersionIPv4, - Description: "Spectrum-X host-to-leaf IP version for guide-based allocation: ipv4 or ipv6. IPv6 is accepted in config but CIDRPool rendering currently supports IPv4 only.", + Description: "Spectrum-X host-to-leaf IP version for guide-based CIDRPool allocation: ipv4 or ipv6.", }, "--topology-file": { Type: "string", diff --git a/pkg/config/default-config.yaml b/pkg/config/default-config.yaml index cc0e05a..67d454a 100644 --- a/pkg/config/default-config.yaml +++ b/pkg/config/default-config.yaml @@ -168,7 +168,7 @@ profile: # multiplaneMode: swplb # CLI parameter (overrides this value): --multiplane-mode (none, swplb, hwplb) # numberOfPlanes: 4 # CLI parameter (overrides this value): --number-of-planes, also used as pfsPerNic # topologyType: 2-tier # CLI parameter (overrides this value): --topology-scheme (2-tier, 3-tier) - # ipVersion: ipv4 # CLI parameter (overrides this value): --ip-version (ipv4, ipv6); CIDRPool rendering currently supports ipv4 + # ipVersion: ipv4 # CLI parameter (overrides this value): --ip-version (ipv4 /31 or ipv6 /64 CIDRPool allocation) # hostFirstOctet: 172 # Config-only IPv4 first octet override; defaults to 172 for 2-tier, 10 for 3-tier # topologyFile: ./topology.json # CLI parameter (overrides this value): --topology-file; spcx-gen/reference-generator or contract-compliant NVIDIA AIR JSON # configMapName: "site-ra23-profile" # Required for RA2.3 when profile is raw data.profile YAML; CLI: --spectrum-x-configmap-name diff --git a/pkg/networkoperatorplugin/spectrumx/addressing.go b/pkg/networkoperatorplugin/spectrumx/addressing.go index 2d4bfea..a335cf5 100644 --- a/pkg/networkoperatorplugin/spectrumx/addressing.go +++ b/pkg/networkoperatorplugin/spectrumx/addressing.go @@ -19,7 +19,7 @@ package spectrumx import ( "encoding/json" "fmt" - "net" + "net/netip" "os" "sort" "strings" @@ -28,10 +28,18 @@ import ( ) type CIDRPool struct { - Name string - CIDR string - Routes []string - StaticAllocations []StaticAllocation + Name string + CIDR string + GatewayIndex int + PerNodeNetworkPrefix int + PerNodeExclusions []PerNodeExclusion + Routes []string + StaticAllocations []StaticAllocation +} + +type PerNodeExclusion struct { + StartIndex int + EndIndex int } type StaticAllocation struct { @@ -137,8 +145,8 @@ type hostLink struct { su int rail int hostIndex int - hostIP net.IP - leafIP net.IP + hostIP netip.Addr + leafIP netip.Addr } type poolKey struct { @@ -163,10 +171,6 @@ func BuildCIDRPools(cfg *config.LaunchKitConfig, group config.ClusterConfig) ([] return nil, nil } spcx := cfg.Profile.SpectrumX - if spcx.IPVersion == config.SpectrumXIPVersionIPv6 { - return nil, fmt.Errorf("Spectrum-X CIDRPool generation currently supports ipVersion=%s only; got %s", - config.SpectrumXIPVersionIPv4, spcx.IPVersion) - } if spcx.TopologyFile == "" { return nil, fmt.Errorf("profile.spectrumX.topologyFile or --topology-file is required for Spectrum-X CIDRPool generation") } @@ -203,16 +207,15 @@ func BuildCIDRPools(cfg *config.LaunchKitConfig, group config.ClusterConfig) ([] } } cidr := poolCIDR(staticAllocations[0].Prefix, spcx) - planeCIDR := planeCIDR(staticAllocations[0].Prefix, spcx) - routes := []string{cidr} - if planeCIDR != "" && planeCIDR != cidr { - routes = append(routes, planeCIDR) - } + settings := poolSettings(spcx) pools = append(pools, CIDRPool{ - Name: poolName(key, group.MergedIdentifier, spcx), - CIDR: cidr, - Routes: routes, - StaticAllocations: staticAllocations, + Name: poolName(key, group.MergedIdentifier, spcx), + CIDR: cidr, + GatewayIndex: settings.gatewayIndex, + PerNodeNetworkPrefix: settings.perNodeNetworkPrefix, + PerNodeExclusions: settings.perNodeExclusions, + Routes: poolRoutes(cidr, spcx), + StaticAllocations: staticAllocations, }) } return pools, nil @@ -284,10 +287,10 @@ func hostLinks(topology *topologyFile, spcx *config.ProfileSpectrumX) ([]hostLin hostIndexes[key+"|"+host.Node] = index } addressPlane := leaf.Attrs.Plane - if spcx.MultiplaneMode == "hwplb" { + if spcx.MultiplaneMode != "swplb" { addressPlane = 0 } - hostIP, leafIP, err := allocateIPv4HostLeaf(spcx, addressPlane, host.Attrs.Rail, host.Attrs.Pod, host.Attrs.SU, index) + hostIP, leafIP, err := allocateHostLeaf(spcx, addressPlane, host.Attrs.Rail, host.Attrs.Pod, host.Attrs.SU, index) if err != nil { return nil, fmt.Errorf("topology host link %d endpoint %s/%s: %w", linkIdx, host.Node, host.Interface, err) } @@ -335,29 +338,80 @@ func countHostIndex(indexes map[string]int, key string) int { return count } -func allocateIPv4HostLeaf(spcx *config.ProfileSpectrumX, plane, rail, pod, su, hostIndex int) (net.IP, net.IP, error) { +func allocateHostLeaf(spcx *config.ProfileSpectrumX, plane, rail, pod, su, hostIndex int) (netip.Addr, netip.Addr, error) { + switch spcx.IPVersion { + case "", config.SpectrumXIPVersionIPv4: + return allocateIPv4HostLeaf(spcx, plane, rail, pod, su, hostIndex) + case config.SpectrumXIPVersionIPv6: + return allocateIPv6HostLeaf(spcx, plane, rail, pod, su, hostIndex) + default: + return netip.Addr{}, netip.Addr{}, fmt.Errorf("unsupported Spectrum-X ipVersion %q", spcx.IPVersion) + } +} + +func allocateIPv4HostLeaf(spcx *config.ProfileSpectrumX, plane, rail, pod, su, hostIndex int) (netip.Addr, netip.Addr, error) { if rail < 0 || rail > 7 { - return nil, nil, fmt.Errorf("rail %d exceeds the supported 3-bit rail field", rail) + return netip.Addr{}, netip.Addr{}, fmt.Errorf("rail %d exceeds the supported 3-bit rail field", rail) } if hostIndex < 0 || hostIndex > 127 { - return nil, nil, fmt.Errorf("host index %d exceeds the supported 7-bit host field", hostIndex) + return netip.Addr{}, netip.Addr{}, fmt.Errorf("host index %d exceeds the supported 7-bit host field", hostIndex) } firstOctet := spcx.HostFirstOctet if firstOctet == 0 { firstOctet = config.SpectrumXDefaultHostFirstOctet(spcx.TopologyType) } if firstOctet < 1 || firstOctet > 255 { - return nil, nil, fmt.Errorf("hostFirstOctet %d must fit in one IPv4 octet", firstOctet) + return netip.Addr{}, netip.Addr{}, fmt.Errorf("hostFirstOctet %d must fit in one IPv4 octet", firstOctet) } second, third, err := ipv4HostLeafMiddleOctets(spcx, plane, rail, pod, su) if err != nil { - return nil, nil, err + return netip.Addr{}, netip.Addr{}, err } - host := net.IPv4(byte(firstOctet), byte(second), byte(third), byte(hostIndex<<1)) - leaf := net.IPv4(byte(firstOctet), byte(second), byte(third), byte(hostIndex<<1|1)) + host := netip.AddrFrom4([4]byte{byte(firstOctet), byte(second), byte(third), byte(hostIndex << 1)}) + leaf := netip.AddrFrom4([4]byte{byte(firstOctet), byte(second), byte(third), byte(hostIndex<<1 | 1)}) return host, leaf, nil } +func allocateIPv6HostLeaf(spcx *config.ProfileSpectrumX, plane, rail, pod, su, hostIndex int) (netip.Addr, netip.Addr, error) { + fields := []struct { + name string + value int + }{ + {name: "plane", value: plane}, + {name: "rail", value: rail}, + {name: "pod", value: pod}, + {name: "su", value: su}, + {name: "host index", value: hostIndex}, + } + for _, field := range fields { + if field.value < 0 || field.value > 255 { + return netip.Addr{}, netip.Addr{}, fmt.Errorf("%s %d exceeds the supported 8-bit field", field.name, field.value) + } + } + + switch spcx.TopologyType { + case config.SpectrumXTopology2Tier: + if pod != 0 { + return netip.Addr{}, netip.Addr{}, fmt.Errorf("pod %d must be zero for 2-tier IPv6 allocation", pod) + } + case config.SpectrumXTopology3Tier: + default: + return netip.Addr{}, netip.Addr{}, fmt.Errorf("unsupported Spectrum-X topologyType %q", spcx.TopologyType) + } + + address := [16]byte{ + 0xfd, 0x02, + 0x00, byte(plane), + byte(rail), byte(pod), + byte(su), byte(hostIndex), + } + hostAddress := address + hostAddress[15] = 1 + leafAddress := address + leafAddress[15] = 2 + return netip.AddrFrom16(hostAddress), netip.AddrFrom16(leafAddress), nil +} + func ipv4HostLeafMiddleOctets(spcx *config.ProfileSpectrumX, plane, rail, pod, su int) (int, int, error) { planeAware := planeAwareAddressing(spcx) switch spcx.TopologyType { @@ -405,6 +459,7 @@ func planeAwareAddressing(spcx *config.ProfileSpectrumX) bool { func allocationsByPool(links []hostLink, allowedNodes map[string]struct{}, spcx *config.ProfileSpectrumX) map[poolKey][]StaticAllocation { result := map[poolKey][]StaticAllocation{} seen := map[poolKey]map[string]struct{}{} + settings := poolSettings(spcx) for _, link := range links { if len(allowedNodes) > 0 { if _, ok := allowedNodes[link.node]; !ok { @@ -422,7 +477,7 @@ func allocationsByPool(links []hostLink, allowedNodes map[string]struct{}, spcx result[key] = append(result[key], StaticAllocation{ Gateway: link.leafIP.String(), NodeName: link.node, - Prefix: link.hostIP.String() + "/31", + Prefix: netip.PrefixFrom(link.hostIP, settings.perNodeNetworkPrefix).Masked().String(), }) } return result @@ -465,11 +520,27 @@ func poolCIDR(prefix string, spcx *config.ProfileSpectrumX) string { return supernet(prefix, railPrefixLength(spcx)) } -func planeCIDR(prefix string, spcx *config.ProfileSpectrumX) string { - return supernet(prefix, planePrefixLength(spcx)) +func poolRoutes(cidr string, spcx *config.ProfileSpectrumX) []string { + if isIPv6(spcx) { + prefixLength := 32 + if spcx.NumberOfPlanes > 1 { + prefixLength = 24 + } + return []string{supernet(cidr, prefixLength)} + } + + planeCIDR := supernet(cidr, planePrefixLength(spcx)) + routes := []string{cidr} + if planeCIDR != "" && planeCIDR != cidr { + routes = append(routes, planeCIDR) + } + return routes } func railPrefixLength(spcx *config.ProfileSpectrumX) int { + if isIPv6(spcx) { + return 40 + } if spcx.TopologyType == config.SpectrumXTopology3Tier { if planeAwareAddressing(spcx) { return 13 @@ -496,13 +567,39 @@ func planePrefixLength(spcx *config.ProfileSpectrumX) int { } func supernet(prefix string, prefixLength int) string { - ip, _, err := net.ParseCIDR(prefix) + parsed, err := netip.ParsePrefix(prefix) if err != nil { return "" } - mask := net.CIDRMask(prefixLength, 32) - network := ip.Mask(mask) - return (&net.IPNet{IP: network, Mask: mask}).String() + if prefixLength < 0 || prefixLength > parsed.Addr().BitLen() { + return "" + } + return netip.PrefixFrom(parsed.Addr(), prefixLength).Masked().String() +} + +type cidrPoolSettings struct { + gatewayIndex int + perNodeNetworkPrefix int + perNodeExclusions []PerNodeExclusion +} + +func poolSettings(spcx *config.ProfileSpectrumX) cidrPoolSettings { + if isIPv6(spcx) { + return cidrPoolSettings{ + gatewayIndex: 2, + perNodeNetworkPrefix: 64, + perNodeExclusions: []PerNodeExclusion{{StartIndex: 2, EndIndex: 2}}, + } + } + return cidrPoolSettings{ + gatewayIndex: 0, + perNodeNetworkPrefix: 31, + perNodeExclusions: []PerNodeExclusion{{StartIndex: 1, EndIndex: 1}}, + } +} + +func isIPv6(spcx *config.ProfileSpectrumX) bool { + return spcx.IPVersion == config.SpectrumXIPVersionIPv6 } func nodeSet(nodes []string) map[string]struct{} { diff --git a/pkg/networkoperatorplugin/spectrumx/addressing_test.go b/pkg/networkoperatorplugin/spectrumx/addressing_test.go index f70c1ff..159c8de 100644 --- a/pkg/networkoperatorplugin/spectrumx/addressing_test.go +++ b/pkg/networkoperatorplugin/spectrumx/addressing_test.go @@ -98,6 +98,9 @@ func TestBuildCIDRPools2TierSWPLB(t *testing.T) { require.Len(t, pools, 4) require.Equal(t, "rail-0-plane-0-gpu-model", pools[0].Name) require.Equal(t, "172.16.0.0/18", pools[0].CIDR) + require.Equal(t, 0, pools[0].GatewayIndex) + require.Equal(t, 31, pools[0].PerNodeNetworkPrefix) + require.Equal(t, []PerNodeExclusion{{StartIndex: 1, EndIndex: 1}}, pools[0].PerNodeExclusions) require.Equal(t, []string{"172.16.0.0/18", "172.16.0.0/14"}, pools[0].Routes) require.Equal(t, []StaticAllocation{ {Gateway: "172.16.0.1", NodeName: "compute-a", Prefix: "172.16.0.0/31"}, @@ -137,8 +140,11 @@ func TestBuildCIDRPools3Tier(t *testing.T) { }) require.NoError(t, err) require.Equal(t, []CIDRPool{{ - Name: "rail-0", - CIDR: "10.0.0.0/13", + Name: "rail-0", + CIDR: "10.0.0.0/13", + GatewayIndex: 0, + PerNodeNetworkPrefix: 31, + PerNodeExclusions: []PerNodeExclusion{{StartIndex: 1, EndIndex: 1}}, Routes: []string{ "10.0.0.0/13", "10.0.0.0/10", @@ -239,9 +245,12 @@ func TestBuildCIDRPoolsFromAIR3Tier(t *testing.T) { }) require.NoError(t, err) require.Equal(t, []CIDRPool{{ - Name: "rail-1", - CIDR: "10.8.0.0/13", - Routes: []string{"10.8.0.0/13", "10.0.0.0/10"}, + Name: "rail-1", + CIDR: "10.8.0.0/13", + GatewayIndex: 0, + PerNodeNetworkPrefix: 31, + PerNodeExclusions: []PerNodeExclusion{{StartIndex: 1, EndIndex: 1}}, + Routes: []string{"10.8.0.0/13", "10.0.0.0/10"}, StaticAllocations: []StaticAllocation{{ Gateway: "10.8.66.7", NodeName: "worker-pod02-su03-h04", @@ -250,17 +259,167 @@ func TestBuildCIDRPoolsFromAIR3Tier(t *testing.T) { }}, pools) } -func TestBuildCIDRPoolsRejectsIPv6UntilRenderable(t *testing.T) { +func TestBuildCIDRPoolsIPv6TwoTierSWPLB(t *testing.T) { + topologyPath := writeTopology(t, `{ + "nodes": [ + {"name": "compute-a", "role": "host", "type": "default"}, + {"name": "compute-b", "role": "host", "type": "default"}, + {"name": "leaf-a", "role": "leaf", "type": "cumulus"} + ], + "links": [ + [ + {"node": "leaf-a", "interface": "swp1s0", "attributes": {"role": "leaf", "plane": 1}}, + {"node": "compute-a", "interface": "eth_p1_r1", "attributes": {"role": "host", "rail": 1, "pod": 0, "su": 3}} + ], + [ + {"node": "leaf-a", "interface": "swp2s0", "attributes": {"role": "leaf", "plane": 1}}, + {"node": "compute-b", "interface": "eth_p1_r1", "attributes": {"role": "host", "rail": 1, "pod": 0, "su": 3}} + ] + ] +}`) cfg := &config.LaunchKitConfig{ Profile: &config.Profile{SpectrumX: &config.ProfileSpectrumX{ - Enable: true, - TopologyType: config.SpectrumXTopology2Tier, - IPVersion: config.SpectrumXIPVersionIPv6, - TopologyFile: "topology.json", + Enable: true, + TopologyType: config.SpectrumXTopology2Tier, + IPVersion: config.SpectrumXIPVersionIPv6, + TopologyFile: topologyPath, + MultiplaneMode: "swplb", + NumberOfPlanes: 2, }}, } - _, err := BuildCIDRPools(cfg, config.ClusterConfig{}) - require.ErrorContains(t, err, "currently supports ipVersion=ipv4 only") + pools, err := BuildCIDRPools(cfg, config.ClusterConfig{WorkerNodes: []string{"compute-a", "compute-b"}}) + require.NoError(t, err) + require.Equal(t, []CIDRPool{{ + Name: "rail-1-plane-1", + CIDR: "fd02:1:100::/40", + GatewayIndex: 2, + PerNodeNetworkPrefix: 64, + PerNodeExclusions: []PerNodeExclusion{{StartIndex: 2, EndIndex: 2}}, + Routes: []string{"fd02::/24"}, + StaticAllocations: []StaticAllocation{ + {Gateway: "fd02:1:100:300::2", NodeName: "compute-a", Prefix: "fd02:1:100:300::/64"}, + {Gateway: "fd02:1:100:301::2", NodeName: "compute-b", Prefix: "fd02:1:100:301::/64"}, + }, + }}, pools) +} + +func TestBuildCIDRPoolsIPv6ThreeTierHWPLB(t *testing.T) { + topologyPath := writeTopology(t, `{ + "nodes": [ + {"name": "compute-a", "role": "host", "type": "default"}, + {"name": "leaf-a", "role": "leaf", "type": "cumulus"} + ], + "links": [[ + {"node": "leaf-a", "interface": "swp1s0", "attributes": {"role": "leaf", "plane": 3}}, + {"node": "compute-a", "interface": "eth_p3_r2", "attributes": {"role": "host", "rail": 2, "pod": 4, "su": 5}} + ]] +}`) + cfg := &config.LaunchKitConfig{ + Profile: &config.Profile{SpectrumX: &config.ProfileSpectrumX{ + Enable: true, + TopologyType: config.SpectrumXTopology3Tier, + IPVersion: config.SpectrumXIPVersionIPv6, + TopologyFile: topologyPath, + MultiplaneMode: "hwplb", + NumberOfPlanes: 4, + }}, + } + pools, err := BuildCIDRPools(cfg, config.ClusterConfig{WorkerNodes: []string{"compute-a"}}) + require.NoError(t, err) + require.Equal(t, []CIDRPool{{ + Name: "rail-2", + CIDR: "fd02:0:200::/40", + GatewayIndex: 2, + PerNodeNetworkPrefix: 64, + PerNodeExclusions: []PerNodeExclusion{{StartIndex: 2, EndIndex: 2}}, + Routes: []string{"fd02::/24"}, + StaticAllocations: []StaticAllocation{{ + Gateway: "fd02:0:204:500::2", + NodeName: "compute-a", + Prefix: "fd02:0:204:500::/64", + }}, + }}, pools) +} + +func TestBuildCIDRPoolsIPv6SinglePlaneRoute(t *testing.T) { + topologyPath := writeSingleLinkTopology(t, "compute-a") + cfg := spectrumXTestConfig(topologyPath, "none") + cfg.Profile.SpectrumX.IPVersion = config.SpectrumXIPVersionIPv6 + cfg.Profile.SpectrumX.NumberOfPlanes = 1 + + pools, err := BuildCIDRPools(cfg, config.ClusterConfig{WorkerNodes: []string{"compute-a"}}) + require.NoError(t, err) + require.Len(t, pools, 1) + require.Equal(t, "fd02::/40", pools[0].CIDR) + require.Equal(t, []string{"fd02::/32"}, pools[0].Routes) +} + +func TestBuildCIDRPoolsFromAIRIPv6(t *testing.T) { + tests := []struct { + name string + mode string + wantPools int + wantLastCIDR string + }{ + {name: "software plane load balancing", mode: "swplb", wantPools: 4, wantLastCIDR: "fd02:3::/40"}, + {name: "hardware plane load balancing", mode: "hwplb", wantPools: 1, wantLastCIDR: "fd02::/40"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + cfg := &config.LaunchKitConfig{ + Profile: &config.Profile{SpectrumX: &config.ProfileSpectrumX{ + Enable: true, + TopologyType: config.SpectrumXTopology2Tier, + IPVersion: config.SpectrumXIPVersionIPv6, + TopologyFile: filepath.Join("testdata", "air-simple-quadplane.json"), + MultiplaneMode: tt.mode, + NumberOfPlanes: 4, + }}, + } + pools, err := BuildCIDRPools(cfg, config.ClusterConfig{ + WorkerNodes: []string{"worker-su01-rack01-h01", "worker-su01-rack01-h02"}, + }) + require.NoError(t, err) + require.Len(t, pools, tt.wantPools) + require.Equal(t, "fd02::/40", pools[0].CIDR) + require.Equal(t, []string{"fd02::/24"}, pools[0].Routes) + require.Equal(t, []StaticAllocation{ + {Gateway: "fd02::2", NodeName: "worker-su01-rack01-h01", Prefix: "fd02::/64"}, + {Gateway: "fd02:0:0:1::2", NodeName: "worker-su01-rack01-h02", Prefix: "fd02:0:0:1::/64"}, + }, pools[0].StaticAllocations) + require.Equal(t, tt.wantLastCIDR, pools[len(pools)-1].CIDR) + }) + } +} + +func TestAllocateIPv6HostLeafRejectsInvalidFields(t *testing.T) { + tests := []struct { + name string + topologyType string + plane int + rail int + pod int + su int + hostIndex int + wantError string + }{ + {name: "negative plane", topologyType: config.SpectrumXTopology3Tier, plane: -1, wantError: "plane -1 exceeds"}, + {name: "rail overflow", topologyType: config.SpectrumXTopology3Tier, rail: 256, wantError: "rail 256 exceeds"}, + {name: "pod overflow", topologyType: config.SpectrumXTopology3Tier, pod: 256, wantError: "pod 256 exceeds"}, + {name: "su overflow", topologyType: config.SpectrumXTopology3Tier, su: 256, wantError: "su 256 exceeds"}, + {name: "host overflow", topologyType: config.SpectrumXTopology3Tier, hostIndex: 256, wantError: "host index 256 exceeds"}, + {name: "two-tier pod", topologyType: config.SpectrumXTopology2Tier, pod: 1, wantError: "pod 1 must be zero"}, + {name: "unsupported topology", topologyType: "fabric", wantError: `unsupported Spectrum-X topologyType "fabric"`}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + spcx := &config.ProfileSpectrumX{TopologyType: tt.topologyType, IPVersion: config.SpectrumXIPVersionIPv6} + _, _, err := allocateIPv6HostLeaf(spcx, tt.plane, tt.rail, tt.pod, tt.su, tt.hostIndex) + require.ErrorContains(t, err, tt.wantError) + }) + } } func TestBuildCIDRPoolsReportsWrongTopologyFile(t *testing.T) { diff --git a/pkg/networkoperatorplugin/sriov_render_test.go b/pkg/networkoperatorplugin/sriov_render_test.go index e2f8bbc..a7851e6 100644 --- a/pkg/networkoperatorplugin/sriov_render_test.go +++ b/pkg/networkoperatorplugin/sriov_render_test.go @@ -245,7 +245,7 @@ func TestProfileManifestsAreValidMultiDocYAML(t *testing.T) { } } -func TestSpectrumXCIDRPoolSeparators(t *testing.T) { +func TestSpectrumXIPv6CIDRPoolRendering(t *testing.T) { profilesUnderTest := []struct { dir string spcxVersion string @@ -287,6 +287,7 @@ func TestSpectrumXCIDRPoolSeparators(t *testing.T) { MultiplaneMode: profile.multiplaneMode, NumberOfPlanes: profile.numberOfPlanes, TopologyType: config.SpectrumXTopology2Tier, + IPVersion: config.SpectrumXIPVersionIPv6, TopologyFile: writeSpectrumXTopology(t, cfg, profile.numberOfPlanes), }, } @@ -309,6 +310,40 @@ func TestSpectrumXCIDRPoolSeparators(t *testing.T) { "%s must have one YAML document per CIDRPool kind", name) for _, doc := range docs { require.Equal(t, "CIDRPool", doc["kind"]) + spec, ok := doc["spec"].(map[string]any) + require.True(t, ok, "CIDRPool has no spec map") + require.EqualValues(t, 2, spec["gatewayIndex"]) + require.EqualValues(t, 64, spec["perNodeNetworkPrefix"]) + require.Contains(t, spec["cidr"], "/40") + + exclusions, ok := spec["perNodeExclusions"].([]any) + require.True(t, ok) + require.Len(t, exclusions, 1) + exclusion, ok := exclusions[0].(map[string]any) + require.True(t, ok) + require.EqualValues(t, 2, exclusion["startIndex"]) + require.EqualValues(t, 2, exclusion["endIndex"]) + + routes, ok := spec["routes"].([]any) + require.True(t, ok) + require.Len(t, routes, 1) + route, ok := routes[0].(map[string]any) + require.True(t, ok) + require.Equal(t, "fd02::/24", route["dst"]) + + allocations, ok := spec["staticAllocations"].([]any) + require.True(t, ok) + require.NotEmpty(t, allocations) + for _, rawAllocation := range allocations { + allocation, ok := rawAllocation.(map[string]any) + require.True(t, ok) + prefix, ok := allocation["prefix"].(string) + require.True(t, ok) + require.True(t, strings.HasSuffix(prefix, "/64")) + gateway, ok := allocation["gateway"].(string) + require.True(t, ok) + require.True(t, strings.HasSuffix(gateway, "::2")) + } } }) } diff --git a/profiles/spectrum-x-ra2.1/60-cidrpool.yaml b/profiles/spectrum-x-ra2.1/60-cidrpool.yaml index 2b8f9fd..c14ffd2 100644 --- a/profiles/spectrum-x-ra2.1/60-cidrpool.yaml +++ b/profiles/spectrum-x-ra2.1/60-cidrpool.yaml @@ -10,11 +10,13 @@ metadata: namespace: "{{$.NetworkOperator.Namespace}}" spec: cidr: {{$pool.CIDR}} - gatewayIndex: 0 - perNodeNetworkPrefix: 31 + gatewayIndex: {{$pool.GatewayIndex}} + perNodeNetworkPrefix: {{$pool.PerNodeNetworkPrefix}} perNodeExclusions: - - startIndex: 1 - endIndex: 1 + {{- range $exclusion := $pool.PerNodeExclusions }} + - startIndex: {{$exclusion.StartIndex}} + endIndex: {{$exclusion.EndIndex}} + {{- end }} routes: {{- range $route := $pool.Routes }} - dst: {{$route}} diff --git a/profiles/spectrum-x-ra2.1/README.md b/profiles/spectrum-x-ra2.1/README.md index f6cb376..b64dec6 100644 --- a/profiles/spectrum-x-ra2.1/README.md +++ b/profiles/spectrum-x-ra2.1/README.md @@ -61,10 +61,11 @@ Same set as the RA2.2 profile: `none`, `swplb`, `hwplb`. | `80-spectrumxrailpoolconfig.yaml` | SpectrumXRailPoolConfig (v1alpha1) | matches 50- | | `90-example-daemonset.yaml` | DaemonSet (example workload) | 1 per group | -`60-cidrpool.yaml` emits `TODO_*` placeholders for `cidr`, `routes`, and -`staticAllocations` — the cluster operator must fill these in before -applying. CIDR ranges come from the switch fabric configuration -(`nv show interface`). +`60-cidrpool.yaml` derives complete IPv4 or IPv6 CIDRPools from the topology +file. IPv4 uses a `/31` per node with gateway index `0`. IPv6 uses the standard +Spectrum-X `/64` per-node layout with leaf gateway `::2`, gateway index `2`, +and a `/40` pool per rail or rail-plane. IPv6 routes are `/32` for one plane +and `/24` for two or four planes. ## NCP differences vs the RA2.2 profile @@ -101,6 +102,8 @@ l8k generate \ --user-config \ --spectrum-x RA2.1 \ --multiplane-mode hwplb --number-of-planes 2 \ + --topology-scheme 2-tier --ip-version ipv6 \ + --topology-file ./topology.json \ --network-operator-release 26.1 \ --save-deployment-files ./deployment ``` diff --git a/profiles/spectrum-x-ra2.2/60-cidrpool.yaml b/profiles/spectrum-x-ra2.2/60-cidrpool.yaml index 2b8f9fd..c14ffd2 100644 --- a/profiles/spectrum-x-ra2.2/60-cidrpool.yaml +++ b/profiles/spectrum-x-ra2.2/60-cidrpool.yaml @@ -10,11 +10,13 @@ metadata: namespace: "{{$.NetworkOperator.Namespace}}" spec: cidr: {{$pool.CIDR}} - gatewayIndex: 0 - perNodeNetworkPrefix: 31 + gatewayIndex: {{$pool.GatewayIndex}} + perNodeNetworkPrefix: {{$pool.PerNodeNetworkPrefix}} perNodeExclusions: - - startIndex: 1 - endIndex: 1 + {{- range $exclusion := $pool.PerNodeExclusions }} + - startIndex: {{$exclusion.StartIndex}} + endIndex: {{$exclusion.EndIndex}} + {{- end }} routes: {{- range $route := $pool.Routes }} - dst: {{$route}} diff --git a/profiles/spectrum-x-ra2.2/README.md b/profiles/spectrum-x-ra2.2/README.md index 07b7c86..c5c0801 100644 --- a/profiles/spectrum-x-ra2.2/README.md +++ b/profiles/spectrum-x-ra2.2/README.md @@ -77,9 +77,11 @@ nodeCapabilities: ### CIDR Pool Configuration -- **cidrPoolPerNodePrefix**: Subnet prefix for per-node allocation (typically `31` for point-to-point) -- **cidrPoolGatewayIndex**: Gateway IP index in the subnet (typically `0`) -- **cidrPoolPerNodeExclusions**: IP indices to exclude from allocation +`--topology-file`, `--topology-scheme`, and `--ip-version` drive CIDRPool +generation. IPv4 uses a `/31` per node with gateway index `0`; IPv6 uses a +`/64` per node with leaf gateway `::2`, gateway index `2`, and a `/40` pool per +rail or rail-plane. IPv6 routes are `/32` for one plane and `/24` for two or +four planes. ## Generated CRDs @@ -101,8 +103,8 @@ The profile generates the following Kubernetes Custom Resources: - Configures Spectrum-X optimized firmware settings (RA2.2). 4. **CIDRPool** (`60-cidrpool.yaml`) - - One pool per rail (non-swplb) or per rail-plane (swplb), with IP placeholders - the cluster operator must fill in. + - One topology-derived IPv4 or IPv6 pool per rail (non-swplb) or per + rail-plane (swplb), including routes and per-node static allocations. 5. **SpectrumXRailPoolConfig** (`80-spectrumxrailpoolconfig.yaml`) - Single `v1alpha2` resource with `railTopology[]`. In swplb, one entry per @@ -149,10 +151,6 @@ spectrumX: rdmaPrefix: "roce_r%rail_id%_p%plane_id%" eSwitchMultiport: "true" bridgeGroupingPolicy: perPF - cidrPoolPerNodePrefix: 31 - cidrPoolGatewayIndex: 0 - cidrPoolPerNodeExclusions: - - "1" profile: fabric: ethernet @@ -163,6 +161,9 @@ profile: spcxVersion: RA2.2 multiplaneMode: hwplb numberOfPlanes: 4 + topologyType: 2-tier + ipVersion: ipv6 + topologyFile: ./topology.json useDRA: false ``` diff --git a/profiles/spectrum-x/60-cidrpool.yaml b/profiles/spectrum-x/60-cidrpool.yaml index 2b8f9fd..c14ffd2 100644 --- a/profiles/spectrum-x/60-cidrpool.yaml +++ b/profiles/spectrum-x/60-cidrpool.yaml @@ -10,11 +10,13 @@ metadata: namespace: "{{$.NetworkOperator.Namespace}}" spec: cidr: {{$pool.CIDR}} - gatewayIndex: 0 - perNodeNetworkPrefix: 31 + gatewayIndex: {{$pool.GatewayIndex}} + perNodeNetworkPrefix: {{$pool.PerNodeNetworkPrefix}} perNodeExclusions: - - startIndex: 1 - endIndex: 1 + {{- range $exclusion := $pool.PerNodeExclusions }} + - startIndex: {{$exclusion.StartIndex}} + endIndex: {{$exclusion.EndIndex}} + {{- end }} routes: {{- range $route := $pool.Routes }} - dst: {{$route}} diff --git a/profiles/spectrum-x/README.md b/profiles/spectrum-x/README.md index 898a656..366fb31 100644 --- a/profiles/spectrum-x/README.md +++ b/profiles/spectrum-x/README.md @@ -85,9 +85,11 @@ nodeCapabilities: ### CIDR Pool Configuration -- **cidrPoolPerNodePrefix**: Subnet prefix for per-node allocation (typically `31` for point-to-point) -- **cidrPoolGatewayIndex**: Gateway IP index in the subnet (typically `0`) -- **cidrPoolPerNodeExclusions**: IP indices to exclude from allocation +`--topology-file`, `--topology-scheme`, and `--ip-version` drive CIDRPool +generation. IPv4 uses a `/31` per node with gateway index `0`; IPv6 uses a +`/64` per node with leaf gateway `::2`, gateway index `2`, and a `/40` pool per +rail or rail-plane. IPv6 routes are `/32` for one plane and `/24` for two or +four planes. ## Generated CRDs @@ -114,8 +116,8 @@ The profile generates the following Kubernetes Custom Resources: `spectrumXOptimized.version` is the generated profile ConfigMap name. 5. **CIDRPool** (`60-cidrpool.yaml`) - - One pool per rail (non-swplb) or per rail-plane (swplb), with IP placeholders - the cluster operator must fill in. + - One topology-derived IPv4 or IPv6 pool per rail (non-swplb) or per + rail-plane (swplb), including routes and per-node static allocations. 6. **SpectrumXRailPoolConfig** (`80-spectrumxrailpoolconfig.yaml`) - Single `v1alpha2` resource with `railTopology[]`. In swplb, one entry per @@ -162,10 +164,6 @@ spectrumX: rdmaPrefix: "roce_r%rail_id%_p%plane_id%" eSwitchMultiport: "true" bridgeGroupingPolicy: perPF - cidrPoolPerNodePrefix: 31 - cidrPoolGatewayIndex: 0 - cidrPoolPerNodeExclusions: - - "1" profile: fabric: ethernet @@ -176,6 +174,9 @@ profile: spcxVersion: RA2.3 multiplaneMode: hwplb numberOfPlanes: 4 + topologyType: 2-tier + ipVersion: ipv6 + topologyFile: ./topology.json configMapName: site-ra23-profile profile: | useSoftwareCCAlgorithm: true diff --git a/skills/k8s-launch-kit-generate/SKILL.md b/skills/k8s-launch-kit-generate/SKILL.md index 9fd0cee..f5057f9 100644 --- a/skills/k8s-launch-kit-generate/SKILL.md +++ b/skills/k8s-launch-kit-generate/SKILL.md @@ -1,6 +1,6 @@ --- name: k8s-launch-kit-generate -version: 1.2.4 +version: 1.2.5 description: "Use this skill when the user wants to generate Kubernetes YAML manifests for NVIDIA networking deployment using k8s-launch-kit (l8k). Activate for: manifest generation, profile selection, choosing between SR-IOV/host-device/RDMA-shared/IPoIB/MacVLAN/Spectrum-X, creating deployment files, or when the user asks 'which profile should I use' or needs help choosing a network configuration." metadata: requires: @@ -35,7 +35,7 @@ to that source file; embedded `--for` generation does not write a config. | `--multiplane-mode` | Auto-defaulted with `--spectrum-x` | `none`, `swplb`, `hwplb` | Auto-defaults from east-west PF deviceID: CX7 / BF3 SuperNIC → `none`, CX8 → `swplb`, CX9 → `hwplb`. Skipped+warned when groups have mixed deviceIDs. | | `--number-of-planes` | Auto-defaulted with `--spectrum-x` | `1`, `2`, `4` | Auto-defaults from deviceID: CX7 / BF3 → 1, CX8 → 2, CX9 → 4. | | `--topology-scheme` | Required with `--spectrum-x` | `2-tier`, `3-tier` | Selects the Spectrum-X topology addressing scheme. | -| `--ip-version` | Required with `--spectrum-x` | `ipv4`, `ipv6` | Selects the address family. CIDRPool rendering currently supports IPv4 only. | +| `--ip-version` | Required with `--spectrum-x` | `ipv4`, `ipv6` | Selects per-node IPv4 `/31` or IPv6 `/64` CIDRPool allocation. | | `--topology-file` | Required with `--spectrum-x` | path | spcx-gen/reference-generator or contract-compliant NVIDIA AIR topology JSON. The format is detected from the JSON structure. | | `--multirail` | Auto-defaulted | — | Auto-defaults to `true`. Explicit `multirail: false` in YAML and `--multirail=false` on the CLI are both preserved. | | `--save-deployment-files` | Yes | — | Output directory for generated YAMLs | @@ -57,7 +57,7 @@ l8k generate --user-config cluster-config.yaml \ # Spectrum-X with hardware plane load balancing l8k generate --user-config cluster-config.yaml \ --spectrum-x RA2.2 --multiplane-mode hwplb --number-of-planes 4 \ - --topology-scheme 2-tier --ip-version ipv4 \ + --topology-scheme 2-tier --ip-version ipv6 \ --topology-file ./topology.json \ --save-deployment-files ./output diff --git a/skills/k8s-launch-kit-generate/references/profiles-summary.md b/skills/k8s-launch-kit-generate/references/profiles-summary.md index 6c106f0..3a91e1f 100644 --- a/skills/k8s-launch-kit-generate/references/profiles-summary.md +++ b/skills/k8s-launch-kit-generate/references/profiles-summary.md @@ -1,6 +1,6 @@ # Profiles Summary -Summary of all 7 l8k profile definitions from `profiles/*/profile.yaml`. +Summary of all 8 l8k profile definitions from `profiles/*/profile.yaml`. Every profile renders the top-level `maintenance` settings into Helm values. For Network Operator 26.1+, those values enable Maintenance Operator, deploy a @@ -81,21 +81,20 @@ direct-drain controllers. - `40-sriovibnetwork.yaml` -- SriovIBNetwork attachment definition - `50-pod.yaml` -## 6. Spectrum-X Multi-Rail (RA2.2, Network Operator 26.4+) +## 6. Spectrum-X Multi-Rail (RA2.3, Network Operator 26.7+) - **Directory**: `profiles/spectrum-x/` - **Plugin**: network-operator - **Requirements**: `fabric=ethernet`, `deployment=sriov`, `multirail=true`, - `spectrumX.spcxVersion=RA2.2`, + `spectrumX.spcxVersion=RA2.3`, `spectrumX.multiplaneMode` in `[swplb, hwplb, none]`, - `minNetworkOperatorRelease=26.4` + `minNetworkOperatorRelease=26.7` - **Node Capabilities**: `sriov: true`, `rdma: true` -- **Description**: Unified Spectrum-X profile covering all three multiplane modes. - Emits a single `SpectrumXRailPoolConfig` (`v1alpha2`) that replaces the - legacy SriovNetworkPoolConfig + SriovNetworkNodePolicy + OVSNetwork trio. - In `swplb` the `railTopology[]` splits each rail into per-plane entries; in - other modes one entry per rail groups all planes. `CIDRPool` manifests are - generated with IP-address placeholders for the operator to fill in. +- **Description**: Unified Spectrum-X profile covering all three multiplane + modes. It deploys the Spectrum-X profile through a ConfigMap and emits one + `SpectrumXRailPoolConfig` (`v1alpha2`). In `swplb`, `railTopology[]` splits + each rail into per-plane entries; in other modes one entry per rail groups + all planes. - **Templates**: - `10-nicclusterpolicy.yaml` -- NicClusterPolicy (with `nicFirmwareStorage` and `spectrumXOperator.xPlane`) @@ -103,15 +102,39 @@ direct-drain controllers. inner `railPciAddresses` list groups all planes of one rail. Applied **before** the NIC config template so firmware settings reference the renamed PFs. - - `30-nicconfigurationtemplate.yaml` -- Spectrum-X firmware settings (RA2.2) + - `28-spectrumxprofile-configmap.yaml` -- ConfigMap-backed RA2.3 profile + - `30-nicconfigurationtemplate.yaml` -- Spectrum-X firmware settings (RA2.3) - `60-cidrpool.yaml` -- One CIDRPool per rail (non-swplb) or per rail-plane - (swplb), with IP placeholders + (swplb), with topology-derived IPv4 or IPv6 static allocations - `80-spectrumxrailpoolconfig.yaml` -- Single SpectrumXRailPoolConfig with `railTopology[]`; omits the removed `spec.withBCM` field because current v1alpha2 strict decoding rejects it + - `85-resourceclaimtemplate.yaml` -- Optional DRA workload claims + - `90-example-daemonset.yaml` -- Example workload + +## 7. Spectrum-X Multi-Rail (RA2.2, Network Operator 26.4) + +- **Directory**: `profiles/spectrum-x-ra2.2/` +- **Plugin**: network-operator +- **Requirements**: `fabric=ethernet`, `deployment=sriov`, `multirail=true`, + `spectrumX.spcxVersion=RA2.2`, + `spectrumX.multiplaneMode` in `[swplb, hwplb, none]`, + `minNetworkOperatorRelease=26.4`, `maxNetworkOperatorRelease=26.4` +- **Node Capabilities**: `sriov: true`, `rdma: true` +- **Description**: RA2.2 variant of the consolidated v1alpha2 profile. In + `swplb`, `railTopology[]` splits each rail into per-plane entries; in other + modes one entry per rail groups all planes. +- **Templates**: + - `10-nicclusterpolicy.yaml` -- NicClusterPolicy with Spectrum-X Operator + - `25-nicinterfacenametemplate.yaml` -- Multi-rail interface naming + - `30-nicconfigurationtemplate.yaml` -- Spectrum-X firmware settings (RA2.2) + - `60-cidrpool.yaml` -- One CIDRPool per rail (non-swplb) or per rail-plane + (swplb), with topology-derived IPv4 or IPv6 static allocations + - `80-spectrumxrailpoolconfig.yaml` -- Single v1alpha2 rail topology resource + - `85-resourceclaimtemplate.yaml` -- Optional DRA workload claims - `90-example-daemonset.yaml` -- Example workload -## 7. Spectrum-X Multi-Rail (RA2.1, Network Operator 26.1) +## 8. Spectrum-X Multi-Rail (RA2.1, Network Operator 26.1) - **Directory**: `profiles/spectrum-x-ra2.1/` - **Plugin**: network-operator @@ -150,7 +173,7 @@ direct-drain controllers. swplb) SriovNetworkNodePolicy with bridge config - `55-ovsnetwork.yaml` -- Matching OVSNetwork with rdma+rail meta-plugins - `60-cidrpool.yaml` -- One CIDRPool per rail (or per rail-plane in - swplb), with IP placeholders + swplb), with topology-derived IPv4 or IPv6 static allocations - `80-spectrumxrailpoolconfig.yaml` -- v1alpha1 glue resource referencing the SR-IOV node policy and CIDR pool - `90-example-daemonset.yaml` -- Example workload diff --git a/skills/k8s-network-engineer/references/profile-decision-tree.md b/skills/k8s-network-engineer/references/profile-decision-tree.md index 97b0ca0..2372897 100644 --- a/skills/k8s-network-engineer/references/profile-decision-tree.md +++ b/skills/k8s-network-engineer/references/profile-decision-tree.md @@ -2,7 +2,7 @@ ## Core Profiles -l8k ships with 7 profile definitions. Each profile is matched by comparing the +l8k ships with 8 profile definitions. Each profile is matched by comparing the user's fabric, deployment type, multirail, and Spectrum-X flags against the profile's `profileRequirements`. @@ -66,24 +66,40 @@ profile's `profileRequirements`. SriovIBNetwork, NicInterfaceNameTemplate, test Pod - **Keywords**: InfiniBand, IB, SR-IOV, HPC, AI training, large-scale -### 6. Spectrum-X Multi-Rail (RA2.2, Network Operator 26.4+) +### 6. Spectrum-X Multi-Rail (RA2.3, Network Operator 26.7+) - **Directory**: `profiles/spectrum-x/` - **Requirements**: `fabric=ethernet`, `deployment=sriov`, `multirail=true`, - `spectrumX.spcxVersion=RA2.2`, `spectrumX.multiplaneMode` in - `[swplb, hwplb, none]`, `minNetworkOperatorRelease=26.4` + `spectrumX.spcxVersion=RA2.3`, `spectrumX.multiplaneMode` in + `[swplb, hwplb, none]`, `minNetworkOperatorRelease=26.7` - **Node capabilities**: `sriov: true`, `rdma: true` - **Use cases**: Multi-tenant AI cloud, Spectrum-X ethernet fabric with OVS hardware offload, BF3 SuperNIC deployments, CX8 with any multiplane mode - **Templates**: NicClusterPolicy (with `nicFirmwareStorage` and `spectrumXOperator.xPlane`), NicInterfaceNameTemplate (applied first), - NicConfigurationTemplate, CIDRPool (one per rail or per rail-plane in - swplb with IP placeholders), SpectrumXRailPoolConfig (`v1alpha2`, single - resource with `railTopology[]` — per-plane entries in swplb, per-rail - entries otherwise, with removed `spec.withBCM` omitted), example DaemonSet -- **Keywords**: Spectrum-X, SPCX, RA2.2, multi-rail, AI cloud, DOCA, swplb, hwplb + profile ConfigMap, NicConfigurationTemplate, topology-derived IPv4 or IPv6 + CIDRPool (one per rail or per rail-plane in swplb), + SpectrumXRailPoolConfig (`v1alpha2`, single resource with `railTopology[]` + and removed `spec.withBCM` omitted), optional ResourceClaimTemplate, + example DaemonSet +- **Keywords**: Spectrum-X, SPCX, RA2.3, multi-rail, AI cloud, DOCA, swplb, hwplb + +### 7. Spectrum-X Multi-Rail (RA2.2, Network Operator 26.4) + +- **Directory**: `profiles/spectrum-x-ra2.2/` +- **Requirements**: `fabric=ethernet`, `deployment=sriov`, `multirail=true`, + `spectrumX.spcxVersion=RA2.2`, `spectrumX.multiplaneMode` in + `[swplb, hwplb, none]`, `minNetworkOperatorRelease=26.4`, + `maxNetworkOperatorRelease=26.4` +- **Node capabilities**: `sriov: true`, `rdma: true` +- **Use cases**: Spectrum-X RA2.2 deployments on Network Operator 26.4 +- **Templates**: NicClusterPolicy, NicInterfaceNameTemplate, + NicConfigurationTemplate, topology-derived IPv4 or IPv6 CIDRPool, + v1alpha2 SpectrumXRailPoolConfig, optional ResourceClaimTemplate, + example DaemonSet +- **Keywords**: Spectrum-X, SPCX, RA2.2, network-operator-26.4, multi-rail, DOCA -### 7. Spectrum-X Multi-Rail (RA2.1, Network Operator 26.1) +### 8. Spectrum-X Multi-Rail (RA2.1, Network Operator 26.1) - **Directory**: `profiles/spectrum-x-ra2.1/` - **Requirements**: `fabric=ethernet`, `deployment=sriov`, `multirail=true`, @@ -112,7 +128,7 @@ profile's `profileRequirements`. - Multiplane mode: **must be `none`** - Number of planes: **must be 1** - Single-plane operation only; no multiplane support in BF3 hardware -- Version: `RA2.1` (with `--network-operator-release 26.1`) or `RA2.2` (26.4+) +- Version: `RA2.1` on Network Operator 26.1, `RA2.2` on 26.4, or `RA2.3` on 26.7+ ### ConnectX-8 (deviceID: 1023) / ConnectX-9 (deviceID: 1025) @@ -121,7 +137,7 @@ profile's `profileRequirements`. - `swplb`: 2 or 4 (default: 4) - `hwplb`: 2 or 4 (default: 4) - `swplb` is the default mode for CX8 when no explicit mode is given -- Version: `RA2.1` (with `--network-operator-release 26.1`) or `RA2.2` (26.4+) +- Version: `RA2.1` on Network Operator 26.1, `RA2.2` on 26.4, or `RA2.3` on 26.7+ ### Multiplane Mode Selection Guide diff --git a/skills/k8s-network-engineer/references/spectrum-x-guide.md b/skills/k8s-network-engineer/references/spectrum-x-guide.md index e87e6ed..dca5d0c 100644 --- a/skills/k8s-network-engineer/references/spectrum-x-guide.md +++ b/skills/k8s-network-engineer/references/spectrum-x-guide.md @@ -112,13 +112,20 @@ ovs-network-plane-1-rail-0 Each rail (or rail-plane combination for swplb) gets its own CIDRPool: -- **perNodeNetworkPrefix**: Typically `31` (point-to-point /31 subnets) -- **gatewayIndex**: `0` (first address in subnet is the gateway) -- **exclusions**: Index `1` is typically excluded +- **IPv4**: Per-node `/31`, gateway index `0`, exclusion index `1` +- **IPv6**: Per-node `/64`, leaf gateway `::2`, gateway index and exclusion + index `2`, and a `/40` pool per rail or rail-plane +- **IPv6 routes**: `/32` for a single plane; `/24` for dual- or quad-plane + deployments - **Size limits**: Each CIDRPool CRD has a 1.5 MB etcd limit - `kubectl apply`: ~6,424 nodes per pool - `kubectl apply --server-side`: ~10,105 nodes per pool (recommended) +IPv6 uses `fd02:00PP:RRDD:SSHH::peer/64`, where the byte-sized fields encode +plane, rail, pod, SU, and host. The host candidate is `::1` and the leaf is +`::2`. `swplb` encodes the plane and creates per-rail-plane pools; `none` and +`hwplb` use address plane zero and create per-rail pools. + ## Generated CRDs The Spectrum-X profile generates these Kubernetes Custom Resources: