Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,17 @@ Discovery fills missing profile values from hardware and built-in defaults.
Values already present under `profile` are preserved, while explicit CLI flags
(`--fabric`, `--deployment-type`, `--multirail`, `--routing`, `--ignore-arp`,
`--spectrum-x`) take precedence.

For Spectrum-X, discovery combines the east-west NIC device ID with each
group's `gpuType` (or `machineType` fallback). H100/H200/B200/GB200 platforms
default to `multiplaneMode: none` and `numberOfPlanes: 1`. B300 and GB300
default to the common GA dual-plane configuration, `swplb` with 2 planes.
Platform type cannot distinguish `swplb` from `hwplb`: both are available on
B300 and GB300, while `hwplb` is an explicit opt-in. Pass
`--multiplane-mode hwplb` when required, and pass `--number-of-planes 4`
explicitly for a quad-plane B300 topology. The same defaults apply when
`--for` supplies hardware from a topology preset.

AI-driven profile selection now lives in the `k8s-launch-kit-*` Claude Code
skills, which wrap the deterministic CLI commands.

Expand Down
4 changes: 2 additions & 2 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,8 @@ Discovery also accepts the profile and Spectrum-X flags below. Explicit flags ov
| Flag | Description |
| --- | --- |
| `--spectrum-x` | Enable Spectrum-X and select RA version, such as `RA2.3`. |
| `--multiplane-mode` | `none`, `swplb`, or `hwplb`. |
| `--number-of-planes` | Plane count for Spectrum-X. |
| `--multiplane-mode` | `none`, `swplb`, or `hwplb`. Defaults from GPU platform and east-west NIC: single-plane H100/H200/B200/GB200 use `none`; B300/GB300 use the GA `swplb` default. Select `hwplb` explicitly. |
| `--number-of-planes` | Plane count for Spectrum-X. Defaults to 1 for single-plane platforms and 2 for B300/GB300; pass 4 explicitly for quad-plane B300. |
| `--topology-scheme` | `2-tier` or `3-tier` for topology-driven CIDRPool allocation. |
| `--ip-version` | `ipv4` for per-node `/31` allocation or `ipv6` for per-node `/64` allocation. |
| `--topology-file` | Path to spcx-gen/reference-generator or contract-compliant NVIDIA AIR topology JSON. The format is detected from its structure. |
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,8 +264,8 @@ profile:
| `ignoreARP` | Adds per-interface ARP tuning outside Spectrum-X. |
| `spectrumX.enable` | Select a Spectrum-X profile. |
| `spectrumX.spcxVersion` | `RA2.1`, `RA2.2`, or `RA2.3`. |
| `spectrumX.multiplaneMode` | `none`, `swplb`, or `hwplb`. |
| `spectrumX.numberOfPlanes` | `1`, `2`, or `4`. |
| `spectrumX.multiplaneMode` | `none`, `swplb`, or `hwplb`. When absent, H100/H200/B200/GB200 default to `none`; B300/GB300 default to the GA `swplb` path. Platform type cannot distinguish `swplb` from `hwplb`, so `hwplb` must be selected explicitly. |
| `spectrumX.numberOfPlanes` | `1`, `2`, or `4`. Single-plane platforms default to 1; B300/GB300 default to 2. Pass 4 explicitly for quad-plane B300. |
| `spectrumX.topologyType` | `2-tier` or `3-tier`. |
| `spectrumX.ipVersion` | `ipv4` for per-node `/31` allocation or `ipv6` for per-node `/64` allocation. |
| `spectrumX.hostFirstOctet` | Config-only first octet for generated IPv4 topology addressing. |
Expand Down
21 changes: 21 additions & 0 deletions docs/user/spectrum-x.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,27 @@ l8k generate \
| `swplb` | Software plane load balancing. Renders per-rail, per-plane resources. |
| `hwplb` | Hardware plane load balancing for larger topologies. |

### Platform-derived defaults

When `multiplaneMode` or `numberOfPlanes` is absent, l8k combines the
discovered GPU platform with the east-west NIC device ID:

| GPU platform | Default mode | Default planes | Notes |
| --- | --- | --- | --- |
| H100, H200, B200, GB200 | `none` | 1 | Single-plane architecture. |
| B300 | `swplb` | 2 | Conservative dual-plane default; pass 4 explicitly for a quad-plane topology. |
| GB300 | `swplb` | 2 | Dual-plane architecture. |

The platform is read from `clusterConfig[].gpuType`, with `machineType` as a
fallback. `--for` presets participate in the same resolution before manifests
are rendered.

B300 and GB300 support both `swplb` and `hwplb`, so platform type does not
identify which load-balancing mechanism the fabric uses. l8k defaults to the
documented GA `swplb` path. Select `hwplb` explicitly when the site topology
requires hardware plane load balancing. Explicit config values and CLI flags
always override these defaults.

```bash
l8k discover \
--spectrum-x RA2.3 \
Expand Down
71 changes: 40 additions & 31 deletions pkg/app/generate.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,36 @@ func (l *Launcher) executeGeneration(configPath string) error {
}
}

// --for supplies the hardware inventory used by profile defaulting as well
// as rendering. Substitute it before resolveProfileSettings so defaults can
// use the preset's machine/gpu types and east-west NICs. Keep the source
// inventory separately: generated preset hardware is transient and must not
// replace clusterConfig in a file-backed user configuration.
sourceClusterConfig := fullConfig.ClusterConfig
presetClusterConfigApplied := false
if l.options.ForPreset != "" {
preset, err := l.presetCatalog.LoadPresetByDir(l.options.ForPreset)
if err != nil {
return apperrors.NewValidationError(
fmt.Sprintf("invalid --for value %q", l.options.ForPreset),
err,
"Run 'l8k preset list' to see available presets",
)
}
selectorMap := parseNodeSelector(l.options.NodeSelector)
cc, synthErr := presets.SynthesizeClusterConfig(l.options.ForPreset, preset, selectorMap)
if synthErr != nil {
return apperrors.NewValidationError(
fmt.Sprintf("preset %q cannot be used with --for", l.options.ForPreset),
synthErr,
"Add a 'capabilities.nodes.{sriov,rdma,ib}' block to the preset's topology.yaml",
)
}
fullConfig.ClusterConfig = []config.ClusterConfig{cc}
presetClusterConfigApplied = true
l.ui.Info("Using preset %q (clusterConfig replaced from preset)", l.options.ForPreset)
}

// Resolve the same defaults/config/CLI precedence that discovery uses
// before either flow persists or consumes the final profile.
if err := l.resolveProfileSettings(fullConfig); err != nil {
Expand Down Expand Up @@ -116,39 +146,18 @@ func (l *Launcher) executeGeneration(configPath string) error {
)
}

// Persist the exact config used for generation: hardware defaults fill
// Persist the exact profile used for generation: hardware defaults fill
// missing fields, existing YAML values survive, and explicit CLI options
// win. Do this before --for substitutes a synthetic clusterConfig so only
// the resolved settings—not preset-only topology—are written back.
if err := l.saveResolvedConfig(configPath, fullConfig, srcConfig, srcConfigYAML); err != nil {
return err
// win. When --for supplied the hardware, restore the source clusterConfig
// only in the write-back copy so preset-only topology remains transient.
resolvedConfigForWriteBack := fullConfig
if presetClusterConfigApplied {
writeBackCopy := *fullConfig
writeBackCopy.ClusterConfig = sourceClusterConfig
resolvedConfigForWriteBack = &writeBackCopy
}

// --for: replace clusterConfig with a synthesized group from a preset.
// This is the explicit ahead-of-time generation path that skips live
// discovery in favor of a static preset description. The CLI layer has
// already enforced --node-selector being set; here we do the substitution
// before the rest of the pipeline runs.
if l.options.ForPreset != "" {
preset, err := l.presetCatalog.LoadPresetByDir(l.options.ForPreset)
if err != nil {
return apperrors.NewValidationError(
fmt.Sprintf("invalid --for value %q", l.options.ForPreset),
err,
"Run 'l8k preset list' to see available presets",
)
}
selectorMap := parseNodeSelector(l.options.NodeSelector)
cc, synthErr := presets.SynthesizeClusterConfig(l.options.ForPreset, preset, selectorMap)
if synthErr != nil {
return apperrors.NewValidationError(
fmt.Sprintf("preset %q cannot be used with --for", l.options.ForPreset),
synthErr,
"Add a 'capabilities.nodes.{sriov,rdma,ib}' block to the preset's topology.yaml",
)
}
fullConfig.ClusterConfig = []config.ClusterConfig{cc}
l.ui.Info("Using preset %q (clusterConfig replaced from preset)", l.options.ForPreset)
if err := l.saveResolvedConfig(configPath, resolvedConfigForWriteBack, srcConfig, srcConfigYAML); err != nil {
return err
}

aggregatedCapabilities := config.AggregateCapabilities(fullConfig.ClusterConfig)
Expand Down
46 changes: 46 additions & 0 deletions pkg/app/generate_profile_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"github.com/nvidia/k8s-launch-kit/pkg/config"
"github.com/nvidia/k8s-launch-kit/pkg/networkoperatorplugin"
"github.com/nvidia/k8s-launch-kit/pkg/options"
"github.com/nvidia/k8s-launch-kit/pkg/presets"
"github.com/nvidia/k8s-launch-kit/pkg/ui"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -121,3 +122,48 @@ func TestResolveSpectrumXTopologyFile(t *testing.T) {
assert.Equal(t, resolvedCLIPath, cfg.Profile.SpectrumX.ResolvedTopologyFile)
})
}

func TestGenerateUsesPresetHardwareForDefaultsWithoutPersistingPresetInventory(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "cluster-config.yaml")
source := `networkOperator:
selectedRelease: "26.4"
profile:
multirail: true
spectrumX:
enable: true
spcxVersion: RA2.2
topologyType: 2-tier
clusterConfig:
- identifier: source-inventory
machineType: source-machine
gpuType: NVIDIA-H200
linkType: Ethernet
pfs:
- deviceID: a2dc
traffic: east-west
`
require.NoError(t, os.WriteFile(configPath, []byte(source), 0o600))

catalog, err := presets.EmbeddedCatalog()
require.NoError(t, err)
launcher := New(options.Options{
ForPreset: "GB300-NVL-NVIDIA-GB300",
NodeSelector: "nvidia.com/gpu.product=NVIDIA-GB300",
})
launcher.ui = ui.NewSilent()
launcher.presetCatalog = catalog

require.NoError(t, launcher.executeGeneration(configPath))

got, err := config.LoadFullConfig(configPath, launcher.logger)
require.NoError(t, err)
require.NotNil(t, got.Profile)
require.NotNil(t, got.Profile.SpectrumX)
assert.Equal(t, "swplb", got.Profile.SpectrumX.MultiplaneMode)
assert.Equal(t, 2, got.Profile.SpectrumX.NumberOfPlanes)
require.Len(t, got.ClusterConfig, 1)
assert.Equal(t, "source-inventory", got.ClusterConfig[0].Identifier)
assert.Equal(t, "NVIDIA-H200", got.ClusterConfig[0].GPUType)
require.Len(t, got.ClusterConfig[0].PFs, 1)
assert.Equal(t, "a2dc", got.ClusterConfig[0].PFs[0].DeviceID)
}
8 changes: 6 additions & 2 deletions pkg/config/default-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,12 @@ profile:
spectrumX: # Spectrum-X configuration (set enable: true or use --spectrum-x CLI flag)
enable: false # CLI parameter (overrides this value): --spectrum-x
# spcxVersion: "RA2.3" # CLI parameter (overrides this value): --spectrum-x
# multiplaneMode: swplb # CLI parameter (overrides this value): --multiplane-mode (none, swplb, hwplb)
# numberOfPlanes: 4 # CLI parameter (overrides this value): --number-of-planes, also used as pfsPerNic
# CLI override: --multiplane-mode. When absent, H100/H200/B200/GB200 use
# none; B300/GB300 use the GA swplb default. Select hwplb explicitly.
# multiplaneMode: swplb
# CLI override: --number-of-planes, also used as pfsPerNic. Single-plane
# platforms use 1; B300/GB300 use 2. Set 4 explicitly for quad-plane B300.
# numberOfPlanes: 2
# topologyType: 2-tier # CLI parameter (overrides this value): --topology-scheme (2-tier, 3-tier)
# ipVersion: ipv4 # CLI parameter (overrides this value): --ip-version (ipv4 /31 or ipv6 /64 CIDRPool allocation)
# hostFirstOctet: 172 # Config-only IPv4 first octet override; defaults to 172 for 2-tier, 10 for 3-tier
Expand Down
4 changes: 2 additions & 2 deletions pkg/options/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ type Options struct {
IgnoreARPSet bool
SpectrumX bool // True when --spectrum-x is set; derived from SPCXVersion != ""
SPCXVersion string // Spectrum-X RA version (the value of --spectrum-x; empty = disabled)
MultiplaneMode string // Spectrum-X multiplane mode (default: swplb)
NumberOfPlanes int // Number of planes for Spectrum-X (default: 4)
MultiplaneMode string // Spectrum-X multiplane mode (platform/NIC-derived when empty)
NumberOfPlanes int // Number of planes for Spectrum-X (platform/NIC-derived when zero)
TopologyScheme string // Spectrum-X topology scheme: 2-tier or 3-tier
IPVersion string // Spectrum-X address family: ipv4 or ipv6
TopologyFile string // Path to spcx-gen/reference-generator or NVIDIA AIR topology JSON for Spectrum-X CIDRPool generation
Expand Down
Loading