diff --git a/README.md b/README.md index 41d573f..7389ac1 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,17 @@ Discovery fills missing profile values from hardware and built-in defaults. Values already present under `profile` are preserved, while explicit CLI flags (`--fabric`, `--deployment-type`, `--multirail`, `--routing`, `--ignore-arp`, `--spectrum-x`) take precedence. + +For Spectrum-X, discovery combines the east-west NIC device ID with each +group's `gpuType` (or `machineType` fallback). H100/H200/B200/GB200 platforms +default to `multiplaneMode: none` and `numberOfPlanes: 1`. B300 and GB300 +default to the common GA dual-plane configuration, `swplb` with 2 planes. +Platform type cannot distinguish `swplb` from `hwplb`: both are available on +B300 and GB300, while `hwplb` is an explicit opt-in. Pass +`--multiplane-mode hwplb` when required, and pass `--number-of-planes 4` +explicitly for a quad-plane B300 topology. The same defaults apply when +`--for` supplies hardware from a topology preset. + AI-driven profile selection now lives in the `k8s-launch-kit-*` Claude Code skills, which wrap the deterministic CLI commands. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index be3dc0b..2a0d134 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -68,8 +68,8 @@ Discovery also accepts the profile and Spectrum-X flags below. Explicit flags ov | Flag | Description | | --- | --- | | `--spectrum-x` | Enable Spectrum-X and select RA version, such as `RA2.3`. | -| `--multiplane-mode` | `none`, `swplb`, or `hwplb`. | -| `--number-of-planes` | Plane count for Spectrum-X. | +| `--multiplane-mode` | `none`, `swplb`, or `hwplb`. Defaults from GPU platform and east-west NIC: single-plane H100/H200/B200/GB200 use `none`; B300/GB300 use the GA `swplb` default. Select `hwplb` explicitly. | +| `--number-of-planes` | Plane count for Spectrum-X. Defaults to 1 for single-plane platforms and 2 for B300/GB300; pass 4 explicitly for quad-plane B300. | | `--topology-scheme` | `2-tier` or `3-tier` for topology-driven CIDRPool allocation. | | `--ip-version` | `ipv4` for per-node `/31` allocation or `ipv6` for per-node `/64` allocation. | | `--topology-file` | Path to spcx-gen/reference-generator or contract-compliant NVIDIA AIR topology JSON. The format is detected from its structure. | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 4401559..5a842ba 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -264,8 +264,8 @@ profile: | `ignoreARP` | Adds per-interface ARP tuning outside Spectrum-X. | | `spectrumX.enable` | Select a Spectrum-X profile. | | `spectrumX.spcxVersion` | `RA2.1`, `RA2.2`, or `RA2.3`. | -| `spectrumX.multiplaneMode` | `none`, `swplb`, or `hwplb`. | -| `spectrumX.numberOfPlanes` | `1`, `2`, or `4`. | +| `spectrumX.multiplaneMode` | `none`, `swplb`, or `hwplb`. When absent, H100/H200/B200/GB200 default to `none`; B300/GB300 default to the GA `swplb` path. Platform type cannot distinguish `swplb` from `hwplb`, so `hwplb` must be selected explicitly. | +| `spectrumX.numberOfPlanes` | `1`, `2`, or `4`. Single-plane platforms default to 1; B300/GB300 default to 2. Pass 4 explicitly for quad-plane B300. | | `spectrumX.topologyType` | `2-tier` or `3-tier`. | | `spectrumX.ipVersion` | `ipv4` for per-node `/31` allocation or `ipv6` for per-node `/64` allocation. | | `spectrumX.hostFirstOctet` | Config-only first octet for generated IPv4 topology addressing. | diff --git a/docs/user/spectrum-x.md b/docs/user/spectrum-x.md index 1acba18..e8d15be 100644 --- a/docs/user/spectrum-x.md +++ b/docs/user/spectrum-x.md @@ -35,6 +35,27 @@ l8k generate \ | `swplb` | Software plane load balancing. Renders per-rail, per-plane resources. | | `hwplb` | Hardware plane load balancing for larger topologies. | +### Platform-derived defaults + +When `multiplaneMode` or `numberOfPlanes` is absent, l8k combines the +discovered GPU platform with the east-west NIC device ID: + +| GPU platform | Default mode | Default planes | Notes | +| --- | --- | --- | --- | +| H100, H200, B200, GB200 | `none` | 1 | Single-plane architecture. | +| B300 | `swplb` | 2 | Conservative dual-plane default; pass 4 explicitly for a quad-plane topology. | +| GB300 | `swplb` | 2 | Dual-plane architecture. | + +The platform is read from `clusterConfig[].gpuType`, with `machineType` as a +fallback. `--for` presets participate in the same resolution before manifests +are rendered. + +B300 and GB300 support both `swplb` and `hwplb`, so platform type does not +identify which load-balancing mechanism the fabric uses. l8k defaults to the +documented GA `swplb` path. Select `hwplb` explicitly when the site topology +requires hardware plane load balancing. Explicit config values and CLI flags +always override these defaults. + ```bash l8k discover \ --spectrum-x RA2.3 \ diff --git a/pkg/app/generate.go b/pkg/app/generate.go index d353ca4..a3560fd 100644 --- a/pkg/app/generate.go +++ b/pkg/app/generate.go @@ -74,6 +74,36 @@ func (l *Launcher) executeGeneration(configPath string) error { } } + // --for supplies the hardware inventory used by profile defaulting as well + // as rendering. Substitute it before resolveProfileSettings so defaults can + // use the preset's machine/gpu types and east-west NICs. Keep the source + // inventory separately: generated preset hardware is transient and must not + // replace clusterConfig in a file-backed user configuration. + sourceClusterConfig := fullConfig.ClusterConfig + presetClusterConfigApplied := false + if l.options.ForPreset != "" { + preset, err := l.presetCatalog.LoadPresetByDir(l.options.ForPreset) + if err != nil { + return apperrors.NewValidationError( + fmt.Sprintf("invalid --for value %q", l.options.ForPreset), + err, + "Run 'l8k preset list' to see available presets", + ) + } + selectorMap := parseNodeSelector(l.options.NodeSelector) + cc, synthErr := presets.SynthesizeClusterConfig(l.options.ForPreset, preset, selectorMap) + if synthErr != nil { + return apperrors.NewValidationError( + fmt.Sprintf("preset %q cannot be used with --for", l.options.ForPreset), + synthErr, + "Add a 'capabilities.nodes.{sriov,rdma,ib}' block to the preset's topology.yaml", + ) + } + fullConfig.ClusterConfig = []config.ClusterConfig{cc} + presetClusterConfigApplied = true + l.ui.Info("Using preset %q (clusterConfig replaced from preset)", l.options.ForPreset) + } + // Resolve the same defaults/config/CLI precedence that discovery uses // before either flow persists or consumes the final profile. if err := l.resolveProfileSettings(fullConfig); err != nil { @@ -116,39 +146,18 @@ func (l *Launcher) executeGeneration(configPath string) error { ) } - // Persist the exact config used for generation: hardware defaults fill + // Persist the exact profile used for generation: hardware defaults fill // missing fields, existing YAML values survive, and explicit CLI options - // win. Do this before --for substitutes a synthetic clusterConfig so only - // the resolved settings—not preset-only topology—are written back. - if err := l.saveResolvedConfig(configPath, fullConfig, srcConfig, srcConfigYAML); err != nil { - return err + // win. When --for supplied the hardware, restore the source clusterConfig + // only in the write-back copy so preset-only topology remains transient. + resolvedConfigForWriteBack := fullConfig + if presetClusterConfigApplied { + writeBackCopy := *fullConfig + writeBackCopy.ClusterConfig = sourceClusterConfig + resolvedConfigForWriteBack = &writeBackCopy } - - // --for: replace clusterConfig with a synthesized group from a preset. - // This is the explicit ahead-of-time generation path that skips live - // discovery in favor of a static preset description. The CLI layer has - // already enforced --node-selector being set; here we do the substitution - // before the rest of the pipeline runs. - if l.options.ForPreset != "" { - preset, err := l.presetCatalog.LoadPresetByDir(l.options.ForPreset) - if err != nil { - return apperrors.NewValidationError( - fmt.Sprintf("invalid --for value %q", l.options.ForPreset), - err, - "Run 'l8k preset list' to see available presets", - ) - } - selectorMap := parseNodeSelector(l.options.NodeSelector) - cc, synthErr := presets.SynthesizeClusterConfig(l.options.ForPreset, preset, selectorMap) - if synthErr != nil { - return apperrors.NewValidationError( - fmt.Sprintf("preset %q cannot be used with --for", l.options.ForPreset), - synthErr, - "Add a 'capabilities.nodes.{sriov,rdma,ib}' block to the preset's topology.yaml", - ) - } - fullConfig.ClusterConfig = []config.ClusterConfig{cc} - l.ui.Info("Using preset %q (clusterConfig replaced from preset)", l.options.ForPreset) + if err := l.saveResolvedConfig(configPath, resolvedConfigForWriteBack, srcConfig, srcConfigYAML); err != nil { + return err } aggregatedCapabilities := config.AggregateCapabilities(fullConfig.ClusterConfig) diff --git a/pkg/app/generate_profile_test.go b/pkg/app/generate_profile_test.go index c6d78a9..e6702ef 100644 --- a/pkg/app/generate_profile_test.go +++ b/pkg/app/generate_profile_test.go @@ -14,6 +14,7 @@ import ( "github.com/nvidia/k8s-launch-kit/pkg/config" "github.com/nvidia/k8s-launch-kit/pkg/networkoperatorplugin" "github.com/nvidia/k8s-launch-kit/pkg/options" + "github.com/nvidia/k8s-launch-kit/pkg/presets" "github.com/nvidia/k8s-launch-kit/pkg/ui" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -121,3 +122,48 @@ func TestResolveSpectrumXTopologyFile(t *testing.T) { assert.Equal(t, resolvedCLIPath, cfg.Profile.SpectrumX.ResolvedTopologyFile) }) } + +func TestGenerateUsesPresetHardwareForDefaultsWithoutPersistingPresetInventory(t *testing.T) { + configPath := filepath.Join(t.TempDir(), "cluster-config.yaml") + source := `networkOperator: + selectedRelease: "26.4" +profile: + multirail: true + spectrumX: + enable: true + spcxVersion: RA2.2 + topologyType: 2-tier +clusterConfig: + - identifier: source-inventory + machineType: source-machine + gpuType: NVIDIA-H200 + linkType: Ethernet + pfs: + - deviceID: a2dc + traffic: east-west +` + require.NoError(t, os.WriteFile(configPath, []byte(source), 0o600)) + + catalog, err := presets.EmbeddedCatalog() + require.NoError(t, err) + launcher := New(options.Options{ + ForPreset: "GB300-NVL-NVIDIA-GB300", + NodeSelector: "nvidia.com/gpu.product=NVIDIA-GB300", + }) + launcher.ui = ui.NewSilent() + launcher.presetCatalog = catalog + + require.NoError(t, launcher.executeGeneration(configPath)) + + got, err := config.LoadFullConfig(configPath, launcher.logger) + require.NoError(t, err) + require.NotNil(t, got.Profile) + require.NotNil(t, got.Profile.SpectrumX) + assert.Equal(t, "swplb", got.Profile.SpectrumX.MultiplaneMode) + assert.Equal(t, 2, got.Profile.SpectrumX.NumberOfPlanes) + require.Len(t, got.ClusterConfig, 1) + assert.Equal(t, "source-inventory", got.ClusterConfig[0].Identifier) + assert.Equal(t, "NVIDIA-H200", got.ClusterConfig[0].GPUType) + require.Len(t, got.ClusterConfig[0].PFs, 1) + assert.Equal(t, "a2dc", got.ClusterConfig[0].PFs[0].DeviceID) +} diff --git a/pkg/config/default-config.yaml b/pkg/config/default-config.yaml index 67d454a..debbd86 100644 --- a/pkg/config/default-config.yaml +++ b/pkg/config/default-config.yaml @@ -165,8 +165,12 @@ profile: spectrumX: # Spectrum-X configuration (set enable: true or use --spectrum-x CLI flag) enable: false # CLI parameter (overrides this value): --spectrum-x # spcxVersion: "RA2.3" # CLI parameter (overrides this value): --spectrum-x - # multiplaneMode: swplb # CLI parameter (overrides this value): --multiplane-mode (none, swplb, hwplb) - # numberOfPlanes: 4 # CLI parameter (overrides this value): --number-of-planes, also used as pfsPerNic + # CLI override: --multiplane-mode. When absent, H100/H200/B200/GB200 use + # none; B300/GB300 use the GA swplb default. Select hwplb explicitly. + # multiplaneMode: swplb + # CLI override: --number-of-planes, also used as pfsPerNic. Single-plane + # platforms use 1; B300/GB300 use 2. Set 4 explicitly for quad-plane B300. + # numberOfPlanes: 2 # topologyType: 2-tier # CLI parameter (overrides this value): --topology-scheme (2-tier, 3-tier) # ipVersion: ipv4 # CLI parameter (overrides this value): --ip-version (ipv4 /31 or ipv6 /64 CIDRPool allocation) # hostFirstOctet: 172 # Config-only IPv4 first octet override; defaults to 172 for 2-tier, 10 for 3-tier diff --git a/pkg/options/options.go b/pkg/options/options.go index 215a542..94c9a41 100644 --- a/pkg/options/options.go +++ b/pkg/options/options.go @@ -72,8 +72,8 @@ type Options struct { IgnoreARPSet bool SpectrumX bool // True when --spectrum-x is set; derived from SPCXVersion != "" SPCXVersion string // Spectrum-X RA version (the value of --spectrum-x; empty = disabled) - MultiplaneMode string // Spectrum-X multiplane mode (default: swplb) - NumberOfPlanes int // Number of planes for Spectrum-X (default: 4) + MultiplaneMode string // Spectrum-X multiplane mode (platform/NIC-derived when empty) + NumberOfPlanes int // Number of planes for Spectrum-X (platform/NIC-derived when zero) TopologyScheme string // Spectrum-X topology scheme: 2-tier or 3-tier IPVersion string // Spectrum-X address family: ipv4 or ipv6 TopologyFile string // Path to spcx-gen/reference-generator or NVIDIA AIR topology JSON for Spectrum-X CIDRPool generation diff --git a/pkg/resolve/defaults.go b/pkg/resolve/defaults.go index eca12a3..7b35c04 100644 --- a/pkg/resolve/defaults.go +++ b/pkg/resolve/defaults.go @@ -23,6 +23,7 @@ package resolve import ( "fmt" "strings" + "unicode" "sigs.k8s.io/controller-runtime/pkg/log" @@ -56,13 +57,17 @@ func (d DefaultDecision) String() string { // --deployment-type ← "sriov" (always). // --multirail ← true (unless config or CLI explicitly sets it). // --routing ← "destination-based" (unless config or CLI sets it). -// --multiplane-mode ← per east-west PF deviceID (only when --spectrum-x): -// 1021 (CX7) / a2dc (BF3 SuperNIC) → "none" -// 1023 (CX8) → "swplb" -// 1025 (CX9) → "hwplb" -// Skipped+warned when groups have mixed deviceIDs. -// --number-of-planes ← per deviceID (only when --spectrum-x): -// 1021 / a2dc → 1, 1023 → 2, 1025 → 4. +// --multiplane-mode ← per GPU platform + east-west PF deviceID +// (only when --spectrum-x): +// H100/H200/B200/GB200 → "none" +// B300/GB300 → "swplb" (the GA default; +// hwplb remains an explicit opt-in) +// Unknown platforms fall back to the NIC family. +// Skipped+warned when groups need different defaults. +// --number-of-planes ← per platform + deviceID (only when +// --spectrum-x): single-plane platforms → 1; +// B300/GB300 → 2 (pass 4 explicitly for a +// quad-plane B300 topology); CX9 fallback → 4. // --network-operator-release ← `config.DefaultSPCXReleaseFor(SPCXVersion)` // (only when --spectrum-x is set). // @@ -151,7 +156,7 @@ func ApplyHardwareDefaults(cfg *config.LaunchKitConfig, opts options.Options) [] // applySpectrumXHardwareDefaults handles the Spectrum-X-only defaults: // implicit fabric/deployment/multirail (forced by Spectrum-X), -// --multiplane-mode, --number-of-planes (from east-west PF deviceID), +// --multiplane-mode, --number-of-planes (from GPU platform + east-west PF), // and --network-operator-release (matched to the chosen RA version). func applySpectrumXHardwareDefaults(cfg *config.LaunchKitConfig, opts options.Options, decisions *[]DefaultDecision) { if cfg.Profile.SpectrumX == nil { @@ -183,32 +188,62 @@ func applySpectrumXHardwareDefaults(cfg *config.LaunchKitConfig, opts options.Op Flag: "--deployment-type", Value: "sriov", Reason: "implied by --spectrum-x", }) } - // --multiplane-mode + --number-of-planes (paired — both come from - // the same deviceID). + // --multiplane-mode + --number-of-planes. An explicit single-plane value + // determines its missing companion without consulting hardware. Otherwise, + // fill the missing values from the platform/NIC pair. modeUnset := cfg.Profile.SpectrumX.MultiplaneMode == "" && opts.MultiplaneMode == "" planesUnset := cfg.Profile.SpectrumX.NumberOfPlanes == 0 && opts.NumberOfPlanes == 0 if modeUnset || planesUnset { - mode, planes, ok, reason := spectrumXDefaultsForDeviceID(cfg.ClusterConfig) + effectiveMode := cfg.Profile.SpectrumX.MultiplaneMode + if opts.MultiplaneMode != "" { + effectiveMode = opts.MultiplaneMode + } + effectivePlanes := cfg.Profile.SpectrumX.NumberOfPlanes + if opts.NumberOfPlanes != 0 { + effectivePlanes = opts.NumberOfPlanes + } + + mode, planes, ok, reason := "", 0, false, "" + needsHardwareMode := modeUnset && effectivePlanes != 1 + needsHardwarePlanes := planesUnset && effectiveMode != "none" + if needsHardwareMode || needsHardwarePlanes { + mode, planes, ok, reason = spectrumXDefaultsForHardware(cfg.ClusterConfig) + } + + modeReason := reason + if modeUnset && effectivePlanes == 1 { + mode = "none" + modeReason = "number-of-planes=1 implies single-plane mode" + } + planesReason := reason + if planesUnset && effectiveMode == "none" { + planes = 1 + planesReason = "multiplane-mode=none implies one plane" + } + + modeResolved := mode != "" && (ok || effectivePlanes == 1) + planesResolved := planes != 0 && (ok || effectiveMode == "none") log.Log.V(1).Info("HW default: --multiplane-mode / --number-of-planes", "groupsConsidered", len(cfg.ClusterConfig), "resolvedMode", mode, "resolvedPlanes", planes, - "applied", ok, "reason", reason) - if ok { - if modeUnset { - cfg.Profile.SpectrumX.MultiplaneMode = mode - *decisions = append(*decisions, DefaultDecision{ - Flag: "--multiplane-mode", Value: mode, - Reason: reason, - }) - } - if planesUnset { - cfg.Profile.SpectrumX.NumberOfPlanes = planes - *decisions = append(*decisions, DefaultDecision{ - Flag: "--number-of-planes", Value: fmt.Sprintf("%d", planes), - Reason: reason, - }) - } - } else { + "modeApplied", modeUnset && modeResolved, + "planesApplied", planesUnset && planesResolved, + "hardwareReason", reason) + if modeUnset && modeResolved { + cfg.Profile.SpectrumX.MultiplaneMode = mode + *decisions = append(*decisions, DefaultDecision{ + Flag: "--multiplane-mode", Value: mode, + Reason: modeReason, + }) + } + if planesUnset && planesResolved { + cfg.Profile.SpectrumX.NumberOfPlanes = planes + *decisions = append(*decisions, DefaultDecision{ + Flag: "--number-of-planes", Value: fmt.Sprintf("%d", planes), + Reason: planesReason, + }) + } + if (modeUnset && !modeResolved) || (planesUnset && !planesResolved) { log.Log.Info("Cannot default --multiplane-mode / --number-of-planes", "reason", reason) } } @@ -272,42 +307,121 @@ func dominantLinkType(groups []config.ClusterConfig) (linkType string, ok bool, return seen, true, "" } -// spectrumXDefaultsForDeviceID returns the Spectrum-X (multiplane-mode, -// number-of-planes) pair that matches the deviceID of the east-west -// PFs across all groups. ok=false when groups have mixed deviceIDs or -// a deviceID isn't in the registered Spectrum-X mapping. -func spectrumXDefaultsForDeviceID(groups []config.ClusterConfig) (mode string, planes int, ok bool, reason string) { +// spectrumXDefaultsForHardware returns a single Spectrum-X +// (multiplane-mode, number-of-planes) pair that is valid for every group. +// GPU platform refines the NIC-family fallback because ConnectX-8 can back +// both single-plane H100/H200/B200/GB200 systems and multiplane B300/GB300 +// systems. Platform does not distinguish swplb from hwplb: both are available +// on B300 and GB300, so l8k defaults to the GA swplb path and requires an +// explicit override for tech-preview hwplb. +func spectrumXDefaultsForHardware(groups []config.ClusterConfig) (mode string, planes int, ok bool, reason string) { if len(groups) == 0 { return "", 0, false, "no clusterConfig groups" } var seenID string + var seenPlatform string + var seenMode string + var seenPlanes int + groupsConsidered := 0 for _, g := range groups { - for _, pf := range g.PFs { - if pf.Traffic != "east-west" { - continue - } - normID := strings.ToLower(pf.DeviceID) - if seenID == "" { - seenID = normID - continue - } - if seenID != normID { - return "", 0, false, fmt.Sprintf("east-west PFs have mixed deviceIDs: %q vs %q", seenID, normID) - } + deviceID, hasEastWest, idReason := eastWestDeviceID(g) + if !hasEastWest { + continue + } + if idReason != "" { + return "", 0, false, idReason } + if seenID == "" { + seenID = deviceID + } else if seenID != deviceID { + return "", 0, false, fmt.Sprintf("east-west PFs have mixed deviceIDs: %q vs %q", seenID, deviceID) + } + + platform := spectrumXGPUPlatform(g) + groupMode, groupPlanes, groupReason := spectrumXDefaultForDeviceAndPlatform(deviceID, platform) + if groupMode == "" { + return "", 0, false, groupReason + } + if groupsConsidered == 0 { + seenMode = groupMode + seenPlanes = groupPlanes + seenPlatform = platform + reason = groupReason + } else if seenMode != groupMode || seenPlanes != groupPlanes { + return "", 0, false, fmt.Sprintf( + "groups require different Spectrum-X defaults: %s/%d for platform %q vs %s/%d for platform %q", + seenMode, seenPlanes, seenPlatform, groupMode, groupPlanes, platform) + } else if seenPlatform != "" && platform != "" && platform != seenPlatform { + reason = fmt.Sprintf("platforms %s and %s share the %s/%d default", seenPlatform, platform, seenMode, seenPlanes) + } + groupsConsidered++ } - if seenID == "" { + if groupsConsidered == 0 { return "", 0, false, "no east-west PFs" } - switch seenID { + return seenMode, seenPlanes, true, reason +} + +func eastWestDeviceID(group config.ClusterConfig) (deviceID string, hasEastWest bool, reason string) { + for _, pf := range group.PFs { + if pf.Traffic != "east-west" { + continue + } + hasEastWest = true + normID := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(pf.DeviceID)), "0x") + if normID == "" { + return "", true, fmt.Sprintf("group %q has an east-west PF without a deviceID", group.Identifier) + } + if deviceID == "" { + deviceID = normID + continue + } + if deviceID != normID { + return "", true, fmt.Sprintf("group %q east-west PFs have mixed deviceIDs: %q vs %q", group.Identifier, deviceID, normID) + } + } + return deviceID, hasEastWest, "" +} + +func spectrumXDefaultForDeviceAndPlatform(deviceID, platform string) (mode string, planes int, reason string) { + switch deviceID { case "1021": - return "none", 1, true, "ConnectX-7 (deviceID 1021)" + return "none", 1, "ConnectX-7 (deviceID 1021)" case "1023": - return "swplb", 2, true, "ConnectX-8 (deviceID 1023)" + switch platform { + case "H100", "H200", "B200", "GB200": + return "none", 1, fmt.Sprintf("%s is a single-plane GPU platform (ConnectX-8 deviceID 1023)", platform) + case "B300": + return "swplb", 2, "B300 conservative dual-plane SWPLB default; pass 4 explicitly for quad-plane" + case "GB300": + return "swplb", 2, "GB300 dual-plane platform; SWPLB is the GA default" + default: + return "swplb", 2, "ConnectX-8 (deviceID 1023) fallback; SWPLB is the GA default" + } case "1025": - return "hwplb", 4, true, "ConnectX-9 (deviceID 1025)" + return "hwplb", 4, "ConnectX-9 (deviceID 1025)" case "a2dc": - return "none", 1, true, "BF3 SuperNIC (deviceID a2dc)" + return "none", 1, "BF3 SuperNIC (deviceID a2dc)" + } + return "", 0, fmt.Sprintf("east-west PF deviceID %q has no Spectrum-X default", deviceID) +} + +func spectrumXGPUPlatform(group config.ClusterConfig) string { + if platform := spectrumXGPUPlatformToken(group.GPUType); platform != "" { + return platform + } + return spectrumXGPUPlatformToken(group.MachineType) +} + +func spectrumXGPUPlatformToken(value string) string { + tokens := strings.FieldsFunc(strings.ToUpper(value), func(r rune) bool { + return !unicode.IsLetter(r) && !unicode.IsDigit(r) + }) + for _, token := range tokens { + switch token { + case "H100", "H200", "B200", "GB200", "B300", "GB300": + return token + } } - return "", 0, false, fmt.Sprintf("east-west PF deviceID %q has no Spectrum-X default", seenID) + return "" } diff --git a/pkg/resolve/defaults_test.go b/pkg/resolve/defaults_test.go index da61b90..2eb6e3f 100644 --- a/pkg/resolve/defaults_test.go +++ b/pkg/resolve/defaults_test.go @@ -133,32 +133,130 @@ func TestApplyHardwareDefaultsRecordsSpectrumXMultirailReason(t *testing.T) { assert.Equal(t, "implied by --spectrum-x", multirailDecisions[0].Reason) } -func TestDefaultSpectrumXModeForDeviceUsesSupportedModes(t *testing.T) { +func TestSpectrumXDefaultsForHardwareUsesPlatformAndNIC(t *testing.T) { tests := []struct { - deviceID string - wantMode string - wantN int + name string + machineType string + gpuType string + deviceID string + wantMode string + wantN int }{ - {deviceID: "1021", wantMode: "none", wantN: 1}, - {deviceID: "a2dc", wantMode: "none", wantN: 1}, - {deviceID: "1023", wantMode: "swplb", wantN: 2}, - {deviceID: "1025", wantMode: "hwplb", wantN: 4}, + {name: "ConnectX-7 fallback", deviceID: "1021", wantMode: "none", wantN: 1}, + {name: "BF3 fallback", deviceID: "a2dc", wantMode: "none", wantN: 1}, + {name: "H100 single plane", gpuType: "NVIDIA-H100-NVL", deviceID: "1023", wantMode: "none", wantN: 1}, + {name: "H200 single plane", gpuType: "NVIDIA-H200", deviceID: "1023", wantMode: "none", wantN: 1}, + {name: "B200 single plane", gpuType: "NVIDIA-B200", deviceID: "1023", wantMode: "none", wantN: 1}, + {name: "GB200 single plane", gpuType: "NVIDIA-GB200", deviceID: "1023", wantMode: "none", wantN: 1}, + {name: "B300 conservative dual plane", gpuType: "NVIDIA-B300", deviceID: "1023", wantMode: "swplb", wantN: 2}, + {name: "GB300 dual plane", gpuType: "NVIDIA-GB300", deviceID: "1023", wantMode: "swplb", wantN: 2}, + {name: "machine type fallback", machineType: "GB300-NVL", deviceID: "0x1023", wantMode: "swplb", wantN: 2}, + {name: "unknown CX8 platform fallback", gpuType: "NVIDIA-GB10", deviceID: "1023", wantMode: "swplb", wantN: 2}, + {name: "ConnectX-9 fallback", deviceID: "1025", wantMode: "hwplb", wantN: 4}, } for _, test := range tests { - t.Run(test.deviceID, func(t *testing.T) { - mode, planes, ok, _ := spectrumXDefaultsForDeviceID([]config.ClusterConfig{ - { - PFs: []config.PFConfig{ - {DeviceID: test.deviceID, Traffic: "east-west"}, - }, - }, + t.Run(test.name, func(t *testing.T) { + mode, planes, ok, reason := spectrumXDefaultsForHardware([]config.ClusterConfig{ + spectrumXTestGroup("group-a", test.machineType, test.gpuType, test.deviceID), }) require.True(t, ok) assert.Equal(t, test.wantMode, mode) assert.Equal(t, test.wantN, planes) assert.Contains(t, config.SupportedMultiplaneModes, mode) + assert.NotEmpty(t, reason) }) } } + +func TestSpectrumXDefaultsForHardwareRejectsConflictingPlatforms(t *testing.T) { + mode, planes, ok, reason := spectrumXDefaultsForHardware([]config.ClusterConfig{ + spectrumXTestGroup("single-plane", "HGX", "NVIDIA-B200", "1023"), + spectrumXTestGroup("dual-plane", "GB300-NVL", "NVIDIA-GB300", "1023"), + }) + + assert.False(t, ok) + assert.Empty(t, mode) + assert.Zero(t, planes) + assert.Contains(t, reason, "different Spectrum-X defaults") +} + +func TestSpectrumXDefaultsForHardwareAcceptsPlatformsWithSameDefault(t *testing.T) { + mode, planes, ok, reason := spectrumXDefaultsForHardware([]config.ClusterConfig{ + spectrumXTestGroup("b300", "HGX-B300", "NVIDIA-B300", "1023"), + spectrumXTestGroup("gb300", "GB300-NVL", "NVIDIA-GB300", "1023"), + }) + + require.True(t, ok) + assert.Equal(t, "swplb", mode) + assert.Equal(t, 2, planes) + assert.Contains(t, reason, "share the swplb/2 default") +} + +func TestApplyHardwareDefaultsCompletesExplicitSinglePlaneSettings(t *testing.T) { + t.Run("mode none implies one plane", func(t *testing.T) { + cfg := &config.LaunchKitConfig{ + Profile: &config.Profile{ + SpectrumX: &config.ProfileSpectrumX{ + Enable: true, + MultiplaneMode: "none", + }, + }, + } + + ApplyHardwareDefaults(cfg, options.Options{}) + + assert.Equal(t, 1, cfg.Profile.SpectrumX.NumberOfPlanes) + }) + + t.Run("one plane implies mode none", func(t *testing.T) { + cfg := &config.LaunchKitConfig{ + Profile: &config.Profile{ + SpectrumX: &config.ProfileSpectrumX{ + Enable: true, + NumberOfPlanes: 1, + }, + }, + } + + ApplyHardwareDefaults(cfg, options.Options{}) + + assert.Equal(t, "none", cfg.Profile.SpectrumX.MultiplaneMode) + }) +} + +func spectrumXTestGroup(identifier, machineType, gpuType, deviceID string) config.ClusterConfig { + rail := 0 + return config.ClusterConfig{ + Identifier: identifier, + MachineType: machineType, + GPUType: gpuType, + LinkType: "Ethernet", + PresetApplied: false, + PresetDeviation: nil, + Capabilities: nil, + PFs: []config.PFConfig{{ + DeviceID: deviceID, + RdmaDevice: "", + PciAddress: "0000:01:00.0", + NetworkInterface: "", + Traffic: "east-west", + Rail: &rail, + PSID: "", + PartNumber: "", + Model: "", + NumaNode: nil, + ConnectedGPU: "", + ConnectedGPUPCIAddress: "", + GPUProximity: "", + }}, + WorkerNodes: nil, + NodeSelector: nil, + ThirdPartyRDMAModules: nil, + StorageModules: nil, + RailPciAddresses: nil, + MergedIdentifier: "", + SourceMachineLabels: nil, + } +} diff --git a/pkg/resolve/validate.go b/pkg/resolve/validate.go index b5b1bcd..2a153e7 100644 --- a/pkg/resolve/validate.go +++ b/pkg/resolve/validate.go @@ -156,11 +156,14 @@ func validateSpectrumXCohort(cfg *config.LaunchKitConfig) error { spcx.IPVersion, config.SupportedSpectrumXIPVersions) } - // Cross-validate mode ↔ planes. "none" collapses to one plane; anything - // else is a contradiction. + // Cross-validate mode ↔ planes. "none" is exactly one plane; software + // and hardware plane load balancing require a multiplane count. if spcx.MultiplaneMode == "none" && spcx.NumberOfPlanes != 1 { return fmt.Errorf("--multiplane-mode none requires --number-of-planes 1, got %d", spcx.NumberOfPlanes) } + if spcx.MultiplaneMode != "none" && spcx.NumberOfPlanes == 1 { + return fmt.Errorf("--multiplane-mode %s requires --number-of-planes 2 or 4, got 1", spcx.MultiplaneMode) + } // Cross-validate (RA version, network-operator-release). allowed := config.SPCXVersionAllowedReleases[spcx.SPCXVersion] diff --git a/pkg/resolve/validate_test.go b/pkg/resolve/validate_test.go index e3331a2..758677c 100644 --- a/pkg/resolve/validate_test.go +++ b/pkg/resolve/validate_test.go @@ -129,3 +129,24 @@ func TestValidateResolvedConfigAcceptsConfigMapProfileForRA23(t *testing.T) { require.NoError(t, ValidateResolvedConfig(cfg)) } + +func TestValidateResolvedConfigRejectsMultiplaneModeWithOnePlane(t *testing.T) { + cfg := &config.LaunchKitConfig{ + NetworkOperator: &config.NetworkOperatorConfig{SelectedRelease: "26.4"}, + Profile: &config.Profile{ + Fabric: "ethernet", + Deployment: "sriov", + Multirail: true, + SpectrumX: &config.ProfileSpectrumX{ + Enable: true, + SPCXVersion: "RA2.2", + MultiplaneMode: "swplb", + NumberOfPlanes: 1, + TopologyType: config.SpectrumXTopology2Tier, + }, + }, + } + + err := ValidateResolvedConfig(cfg) + require.ErrorContains(t, err, "--multiplane-mode swplb requires --number-of-planes 2 or 4") +} diff --git a/skills/k8s-launch-kit-config/references/config-reference.md b/skills/k8s-launch-kit-config/references/config-reference.md index 5217880..4d59ade 100644 --- a/skills/k8s-launch-kit-config/references/config-reference.md +++ b/skills/k8s-launch-kit-config/references/config-reference.md @@ -319,16 +319,18 @@ profile: # value of --spectrum-x on the CLI. spcxVersion: "RA2.2" - # string | default: derived from east-west NIC device ID + # string | default: derived from GPU platform + east-west NIC device ID # Multiplane mode: "none", "swplb" (software PLB), or "hwplb" # (hardware PLB). When Spectrum-X is enabled and this is absent, - # discovery derives it from the east-west NIC device ID. + # H100/H200/B200/GB200 default to none; B300/GB300 default to the + # GA swplb path. Platform type cannot select hwplb; override explicitly. multiplaneMode: swplb - # int | default: derived from east-west NIC device ID + # int | default: derived from GPU platform + east-west NIC device ID # Number of network planes (1, 2, or 4). Also used as pfsPerNic for - # Spectrum-X. When absent, discovery derives it with multiplaneMode. - numberOfPlanes: 4 + # Spectrum-X. Single-plane platforms default to 1; B300/GB300 default + # to 2. Set 4 explicitly for a quad-plane B300 topology. + numberOfPlanes: 2 # ============================================================================ # Validation Configuration diff --git a/skills/k8s-launch-kit-generate/SKILL.md b/skills/k8s-launch-kit-generate/SKILL.md index f5057f9..cb20819 100644 --- a/skills/k8s-launch-kit-generate/SKILL.md +++ b/skills/k8s-launch-kit-generate/SKILL.md @@ -32,8 +32,8 @@ to that source file; embedded `--for` generation does not write a config. | `--fabric` | Auto-defaulted | `ethernet`, `infiniband` | Network fabric. Auto-defaults from the cluster's unanimous `linkType` when omitted (Unit 5 fabric probe); skipped+warned when groups disagree or any has unverified linkType. | | `--deployment-type` | Auto-defaulted | `sriov`, `rdma_shared`, `host_device` | Deployment type. Auto-defaults to `sriov`. | | `--spectrum-x` | — | `RA2.1`, `RA2.2`, `RA2.3` | Enable Spectrum-X profile by passing the SPC-X RA version. Implies ethernet fabric, sriov deployment, and multirail. | -| `--multiplane-mode` | Auto-defaulted with `--spectrum-x` | `none`, `swplb`, `hwplb` | Auto-defaults from east-west PF deviceID: CX7 / BF3 SuperNIC → `none`, CX8 → `swplb`, CX9 → `hwplb`. Skipped+warned when groups have mixed deviceIDs. | -| `--number-of-planes` | Auto-defaulted with `--spectrum-x` | `1`, `2`, `4` | Auto-defaults from deviceID: CX7 / BF3 → 1, CX8 → 2, CX9 → 4. | +| `--multiplane-mode` | Auto-defaulted with `--spectrum-x` | `none`, `swplb`, `hwplb` | Auto-defaults from GPU platform plus east-west PF deviceID: H100/H200/B200/GB200 → `none`; B300/GB300 → the GA `swplb` path. Platform cannot identify `hwplb`; select it explicitly. Unknown platforms fall back to NIC family. | +| `--number-of-planes` | Auto-defaulted with `--spectrum-x` | `1`, `2`, `4` | Single-plane platforms → 1; B300/GB300 → 2. Pass 4 explicitly for quad-plane B300. An explicit `none` also implies 1, and an explicit 1 implies `none`. | | `--topology-scheme` | Required with `--spectrum-x` | `2-tier`, `3-tier` | Selects the Spectrum-X topology addressing scheme. | | `--ip-version` | Required with `--spectrum-x` | `ipv4`, `ipv6` | Selects per-node IPv4 `/31` or IPv6 `/64` CIDRPool allocation. | | `--topology-file` | Required with `--spectrum-x` | path | spcx-gen/reference-generator or contract-compliant NVIDIA AIR topology JSON. The format is detected from the JSON structure. | @@ -154,7 +154,10 @@ win when a one-off override is needed. ## Tips - Default to SR-IOV Ethernet for new GPU cluster deployments unless told otherwise. -- For Spectrum-X, NIC type determines available multiplane modes — read `references/spectrum-x-guide.md`. +- For Spectrum-X, GPU platform and NIC type determine the safe defaults, but + B300/GB300 platform type does not distinguish SWPLB from HWPLB. l8k defaults + to SWPLB; use an explicit HWPLB override when the site topology requires it. + Read `references/spectrum-x-modes.md`. - NVIDIA AIR topology support requires the documented one-based node/interface naming contract (`su`, `h`, `leaf-p

`, `r`, `railp

`, and `pod` for 3-tier). See `docs/user/spectrum-x.md` in the l8k repository. diff --git a/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md b/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md index baf1e1e..d463e6a 100644 --- a/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md +++ b/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md @@ -15,7 +15,8 @@ All Spectrum-X deployments require: ## Mode: none -- **NIC type**: BlueField-3 SuperNIC only (deviceID `a2dc`) +- **NIC type**: BlueField-3 SuperNIC (`a2dc`), ConnectX-7 (`1021`), or + ConnectX-8 (`1023`) in a single-plane platform - **Number of planes**: 1 (fixed, no other value allowed) - **Profile**: `spectrum-x` (the base Spectrum-X profile) - **Resource naming**: Per-rail only @@ -27,14 +28,13 @@ ovs-network-rail-0 ovs-network-rail-1 ``` -- **Description**: Single-plane operation for BF3 SuperNICs. No multiplane support - because BF3 hardware does not implement plane load balancing. This is the only - valid mode for BF3 deployments. +- **Description**: Single-plane operation. This is the only valid mode for BF3 + and CX7, and the default for H100/H200/B200/GB200 GPU platforms. ## Mode: swplb (Software Plane Load Balancing) - **NIC type**: ConnectX-8 (deviceID `1023`) or ConnectX-9 (deviceID `1025`) -- **Number of planes**: 2 or 4 (default: 4) +- **Number of planes**: 2 or 4; l8k defaults B300/GB300 to 2 - **Profile**: `spectrum-x` (unified profile, branches on swplb internally) - **Resource naming**: Per-rail AND per-plane (finest granularity) @@ -49,14 +49,14 @@ ovs-network-plane-1-rail-0 - **Description**: Software-based distribution of traffic across multiple planes. Each rail-plane combination gets its own SR-IOV policy, OVS network, and CIDR - pool. This provides the finest resource granularity and is the default mode for - CX8 deployments. Best for small-to-medium Spectrum-X clusters. + pool. This provides the finest resource granularity and is the GA default for + B300/GB300 deployments. Best for small-to-medium Spectrum-X clusters. - **docaEswitchMax**: planes x number of rails ## Mode: hwplb (Hardware Plane Load Balancing) - **NIC type**: ConnectX-8 (deviceID `1023`) or ConnectX-9 (deviceID `1025`) -- **Number of planes**: 2 or 4 (default: 4) +- **Number of planes**: 2 or 4, selected explicitly with `hwplb` - **Profile**: `spectrum-x` (base Spectrum-X profile) - **Resource naming**: Per-rail only (hardware handles plane distribution) @@ -82,13 +82,27 @@ ovs-network-rail-1 | ConnectX-8 | `1023` | `swplb`, `hwplb` | `swplb` | | ConnectX-9 | `1025` | `swplb`, `hwplb` | `hwplb` | +NIC type only constrains what is possible. Platform type does not distinguish +`swplb` from `hwplb` on B300 or GB300 because both modes support both +platforms. For the documented CX8/B300/GB300 combinations, Launch Kit uses +`swplb` as the GA default and treats `hwplb` as an explicit site-topology +choice. Unknown platforms retain the NIC-family fallback in the table above. + +## Platform Default Summary + +| GPU platform | Default mode | Default planes | Notes | +|--------------|--------------|----------------|-------| +| H100/H200/B200/GB200 | `none` | 1 | Single-plane architecture | +| B300 | `swplb` | 2 | Pass 4 explicitly for quad-plane | +| GB300 | `swplb` | 2 | Dual-plane architecture | + ## Number of Planes Rules | Mode | Valid Values | Default | Notes | |-----------|-------------|---------|----------------------------------------| | `none` | 1 | 1 | CX7/BF3, single plane | -| `swplb` | 2, 4 | 4 | More planes = finer resource granularity| -| `hwplb` | 2, 4 | 4 | More planes = more hardware capacity | +| `swplb` | 2, 4 | 2 on B300/GB300 | Pass 4 explicitly for quad-plane B300 | +| `hwplb` | 2, 4 | explicit | Platform type cannot select the mode | ## Version @@ -117,7 +131,7 @@ decoding. | CX7 deployment | `none` | | CX8, small cluster, fine-grained control | `swplb` | | CX8, large multi-tier topology | `hwplb` | -| Not sure (CX8) | `swplb` (default)| +| Not sure (B300/GB300) | `swplb` (GA default) | ## Validation Rules diff --git a/skills/k8s-network-engineer/references/config-schema.md b/skills/k8s-network-engineer/references/config-schema.md index 5528967..0b122eb 100644 --- a/skills/k8s-network-engineer/references/config-schema.md +++ b/skills/k8s-network-engineer/references/config-schema.md @@ -220,8 +220,8 @@ the single CLI gateway: a non-empty value sets `enable: true` AND populates |------------------|--------|----------|------------------------------------------------------|----------------------------------------| | `enable` | bool | `false` | derived from `--spectrum-x` (true when value is set) | Enable Spectrum-X profile | | `spcxVersion` | string | `RA2.2` | value of `--spectrum-x` | Spectrum-X RA version. `RA2.2` (Network Operator 26.4+) or `RA2.1` (26.1 only). | -| `multiplaneMode` | string | `swplb` | `--multiplane-mode` | `swplb`, `hwplb`, or `none`| -| `numberOfPlanes` | int | `4` | `--number-of-planes` | 1, 2, or 4 (also used as pfsPerNic) | +| `multiplaneMode` | string | platform/NIC-derived | `--multiplane-mode` | H100/H200/B200/GB200 use `none`; B300/GB300 use GA `swplb`; select `hwplb` explicitly | +| `numberOfPlanes` | int | platform/NIC-derived | `--number-of-planes` | Single-plane platforms use 1; B300/GB300 use 2; pass 4 explicitly for quad-plane B300 | CLI flags always override config file values for all profile fields. diff --git a/skills/k8s-network-engineer/references/glossary.md b/skills/k8s-network-engineer/references/glossary.md index 73c8340..1d590f6 100644 --- a/skills/k8s-network-engineer/references/glossary.md +++ b/skills/k8s-network-engineer/references/glossary.md @@ -27,8 +27,8 @@ ## Spectrum-X - **Multiplane mode**: How Spectrum-X organizes switch fabric planes. Options: `none` (CX7/BF3), `swplb` (software plane load balancing), and `hwplb` (hardware plane load balancing). -- **SWPLB (Software Plane Load Balancing)**: Each rail-plane combination gets its own set of Kubernetes resources. Default for CX8. -- **HWPLB (Hardware Plane Load Balancing)**: Planes managed by switch hardware. Resources generated per-rail only. For larger-scale 2/3-tier topologies. +- **SWPLB (Software Plane Load Balancing)**: Each rail-plane combination gets its own set of Kubernetes resources. The GA Launch Kit default for B300/GB300. +- **HWPLB (Hardware Plane Load Balancing)**: Planes managed by switch hardware. Resources generated per-rail only. For larger-scale 2/3-tier topologies; selected explicitly because B300/GB300 platform type does not distinguish it from SWPLB. ## Network Operator Components diff --git a/skills/k8s-network-engineer/references/profile-decision-tree.md b/skills/k8s-network-engineer/references/profile-decision-tree.md index 2372897..fd453a8 100644 --- a/skills/k8s-network-engineer/references/profile-decision-tree.md +++ b/skills/k8s-network-engineer/references/profile-decision-tree.md @@ -134,17 +134,21 @@ profile's `profileRequirements`. - Multiplane modes: `swplb` or `hwplb` - Number of planes: - - `swplb`: 2 or 4 (default: 4) - - `hwplb`: 2 or 4 (default: 4) -- `swplb` is the default mode for CX8 when no explicit mode is given + - `swplb`: 2 or 4 (B300/GB300 default: 2) + - `hwplb`: 2 or 4 (explicit site-topology choice) +- H100/H200/B200/GB200 default to single-plane `none` / 1 even when the + east-west NIC is CX8. +- B300/GB300 default to `swplb` / 2. Pass 4 explicitly for quad-plane B300. +- Platform type cannot distinguish `swplb` from `hwplb`; both are supported on + B300 and GB300, so `hwplb` must be selected explicitly. - Version: `RA2.1` on Network Operator 26.1, `RA2.2` on 26.4, or `RA2.3` on 26.7+ ### Multiplane Mode Selection Guide | Mode | NIC | Scale | Resources | Use When | |----------|------|----------------|--------------------------|-----------------------------------| -| `none` | BF3 | Any | Per-rail | BF3 SuperNIC deployments | -| `swplb` | CX8 | Small-medium | Per-rail-per-plane | Default for CX8, finer granularity| +| `none` | BF3/CX7/CX8 | Any | Per-rail | Single-plane GPU platforms | +| `swplb` | CX8 | Small-medium | Per-rail-per-plane | GA default for B300/GB300 | | `hwplb` | CX8 | Large (2/3-tier)| Per-rail only | Large-scale multi-tier topologies | ### Number of Planes Rules @@ -152,8 +156,8 @@ profile's `profileRequirements`. | Mode | Valid Values | Default | Notes | |-----------|-------------|---------|--------------------------------| | `none` | 1 | 1 | CX7/BF3, no planes | -| `swplb` | 2, 4 | 4 | More planes = more granularity | -| `hwplb` | 2, 4 | 4 | More planes = more capacity | +| `swplb` | 2, 4 | 2 on B300/GB300 | Pass 4 explicitly for quad-plane B300 | +| `hwplb` | 2, 4 | explicit | Select from site topology, not platform type | ## Keyword Matching Heuristics diff --git a/skills/k8s-network-engineer/references/spectrum-x-guide.md b/skills/k8s-network-engineer/references/spectrum-x-guide.md index dca5d0c..7ac21eb 100644 --- a/skills/k8s-network-engineer/references/spectrum-x-guide.md +++ b/skills/k8s-network-engineer/references/spectrum-x-guide.md @@ -11,10 +11,11 @@ It always requires `fabric=ethernet`, `deployment=sriov`, and `multirail=true`. Spectrum-X supports three multiplane modes that determine how network planes are organized and how resources are named: -### none (BF3 Only) +### none (Single Plane) - Single plane, no multiplane support -- BlueField-3 SuperNIC only (deviceID `a2dc`) +- BlueField-3 SuperNIC (`a2dc`), ConnectX-7 (`1021`), or ConnectX-8 + (`1023`) on H100/H200/B200/GB200 - Number of planes: 1 (fixed) - Resources are named per-rail only - Simplest Spectrum-X deployment @@ -23,7 +24,7 @@ organized and how resources are named: - ConnectX-8 (deviceID `1023`) or ConnectX-9 (deviceID `1025`) - Software-based load balancing across planes -- Number of planes: 2 or 4 (default: 4) +- Number of planes: 2 or 4 (B300/GB300 default: 2) - Resources are named per-rail AND per-plane (finest granularity) - SpectrumXRailPoolConfig emits one `railTopology[]` entry per rail-plane, each with its own `cidrPoolRef: rail-{i}-plane-{p}` @@ -33,10 +34,15 @@ organized and how resources are named: - ConnectX-8 (deviceID `1023`) or ConnectX-9 (deviceID `1025`) - Hardware-based load balancing across planes -- Number of planes: 2 or 4 (default: 4) +- Number of planes: 2 or 4 (explicit site-topology choice) - Resources are named per-rail only (hardware handles plane distribution) - Better for large-scale 2-tier and 3-tier network topologies +Both B300 and GB300 support `swplb` and `hwplb`, so the GPU platform cannot +identify which load-balancing mechanism the fabric uses. Launch Kit defaults +these platforms to the GA `swplb` path with 2 planes. Select `hwplb` +explicitly, and pass 4 explicitly for a quad-plane B300 topology. + All three modes are supported by the Spectrum-X profiles. Pick the profile by the value of `--spectrum-x` (the legacy `--spcx-version` has been folded into `--spectrum-x`): @@ -163,11 +169,11 @@ l8k --user-config config.yaml \ --multirail --spectrum-x \ --multiplane-mode none --number-of-planes 1 -# CX8 with software plane load balancing (default) +# B300/GB300 with software plane load balancing (default) l8k --user-config config.yaml \ --fabric ethernet --deployment-type sriov \ --multirail --spectrum-x \ - --multiplane-mode swplb --number-of-planes 4 + --multiplane-mode swplb --number-of-planes 2 # CX8 with hardware plane load balancing (large scale) l8k --user-config config.yaml \