Skip to content

feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task#8096

Open
rchincha wants to merge 70 commits into
mainfrom
origin/rchinchani/rcv1p-2
Open

feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task#8096
rchincha wants to merge 70 commits into
mainfrom
origin/rchinchani/rcv1p-2

Conversation

@rchincha
Copy link
Copy Markdown

@rchincha rchincha commented Mar 16, 2026 •
edited
Loading

What this PR does / why we need it

This PR implements RCV1P (Robust Certificate Validation for 1P) — the next-generation mechanism for distributing Azure root CA certificates to AKS nodes. Instead of hardcoding certificate bundles, RCV1P queries the Azure wireserver at provisioning time to download and install the latest root certificates into the OS trust store.

Reference: https://eng.ms/docs/products/onecert-certificates-key-vault-and-dsms/onecert-customer-guide/autorotationandecr/rcv1ptsg

Summary of Changes

1. Linux: Unified cert bootstrap flow (init-aks-custom-cloud.sh)

  • Consolidated 3 scripts into 1: Removed init-aks-custom-cloud-mariner.sh, init-aks-custom-cloud-operation-requests-mariner.sh, and init-aks-custom-cloud-operation-requests.sh. All cert logic now flows through a single init-aks-custom-cloud.sh that detects the distro (Ubuntu, Mariner, AzureLinux, Flatcar, ACL) at runtime.
  • Extracted repo depot logic: Moved Ubuntu repo depot initialization and chrony configuration into a new init-aks-custom-cloud-repos.sh to keep base customData size small for non-custom-cloud scenarios (critical for Flatcar/ACL which have tight size limits).
  • Two cert endpoint modes: legacy (ussec/usnat regions) and rcv1p (all other regions), selected by cloud location at runtime.
  • Fatal failure handling: Wireserver cert retrieval failures are fatal on both Linux and Windows — after exhausting retries, provisioning fails rather than continuing without certificates. The cert mode (legacy vs rcv1p) is always determined and must succeed.

2. Windows: CA cert refresh task and rcv1p support (kubernetesfunc.ps1)

  • New Get-CACertificates with -Location parameter: Determines cert endpoint mode from location, uses legacy endpoint for ussec/usnat, rcv1p for all others.
  • New Register-CACertificatesRefreshTask: Registers a daily scheduled task to refresh CA certificates, with backward compatibility for older VHDs that don't accept -Location.
  • New Should-InstallCACertificatesRefreshTask: Gates refresh task registration on wireserver opt-in status.
  • Pester tests: 260 lines of tests covering URI construction, endpoint mode selection, and refresh task gating.

3. E2E tests (e2e/scenario_rcv1p_test.go, e2e/scenario_rcv1p_win_test.go)

  • Linux tests: Ubuntu 2204, Ubuntu 2404, AzureLinux V3, Flatcar, ACL — each validates cert download, trust store installation, and refresh schedule.
  • Windows tests: Windows 2022, 23H2, 2025 — validates cert download to C:\ca, Windows certificate store import, and scheduled task registration.
  • Negative test: Test_RCV1P_NotOptedIn verifies that omitting the VM opt-in tag correctly prevents cert installation.
  • Dedicated pipeline: .pipelines/e2e-rcv1p.yaml runs daily at 3am PST with tag filter rcv1pcertmode=true (not yet enabled).

4. E2E infrastructure: multi-subscription and VM instance tagging

  • Multi-subscription support: RCV1P tests run in a dedicated subscription (RCV1P_SUBSCRIPTION_ID) with the Microsoft.Compute/PlatformSettingsOverride feature flag. Added SubscriptionID field to scenarios and GetAzure()/GetSubscriptionID() helpers.
  • VMSS opt-in tagging: Sets the wireserver opt-in tag (platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) on the VMSS at creation time via a VMConfigMutator. VMSS-level tags inherit to VM instances automatically.

⚠️ Critical Design Decisions

1. Cert endpoint mode is determined by cloud location, not a flag

Decision: ussec*/usnat* → legacy mode, everything else → rcv1p mode. This is determined at runtime from the node's Azure location.
Why: Avoids requiring a new API contract field. The location-based approach lets us roll out rcv1p incrementally — ussec/usnat stay on the legacy endpoint that works today, while all other regions use the new rcv1p endpoint with opt-in gating.

2. Two-layer access control for rcv1p

Decision: Both conditions must be met for cert installation:

  1. Subscription feature flag (Microsoft.Compute/PlatformSettingsOverride) enables the wireserver endpoint
  2. VM instance tag (platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) grants per-VM access

Why: Defense in depth — the subscription flag is a coarse gate, the VM tag provides per-node opt-in control. Without the tag, wireserver returns IsOptedInForRootCerts=false.

3. VM opt-in tag is set at VMSS creation time

Decision: The opt-in tag (platformsettings.host_environment.service.platform_optedin_for_rootcerts=true) is set on the VMSS at creation time and inherits to all VM instances automatically.
Why: VMSS-level tags propagate to VM instances, and wireserver reads the tag from the VM instance to determine opt-in status. In E2E tests, the positive tests set the tag via a VMConfigMutator at VMSS creation, while the negative test (Test_RCV1P_NotOptedIn) simply omits the tag to verify wireserver returns IsOptedInForRootCerts=false.

4. Get-CACertificates moved outside IsAKSCustomCloud guard (Windows)

Decision: Get-CACertificates -Location $Location -FailOnError now runs for all clouds, not just custom clouds.
Why: RCV1P applies to all clouds. The function itself handles the location-based mode selection internally and gracefully skips cert installation when wireserver returns IsOptedInForRootCerts=false (which is the case on public cloud without the feature flag).

5. Wireserver failures are fatal after retries

Decision: If wireserver cert endpoints fail after exhausting retries, provisioning fails (exit 1 on Linux, throw on Windows with -FailOnError).
Why: Cert installation is required for the selected mode. Silently continuing without certificates would leave the node in an inconsistent state. Retries with backoff handle transient wireserver issues (rate limiting, temporary unavailability).

6. Backward compatibility for Windows VHD/CSE version skew

Decision: kuberneteswindowssetup.ps1 guards Register-CACertificatesRefreshTask with Get-Command checks before calling it.
Why: Windows VHD and CSE release independently. Newer CSE must not crash on older VHDs that don't have these functions. The guard falls back gracefully.

Testing Evidence

MSFT tenant (default E2E subscription)

Linux (Build 158446017):

  • All distros (Ubuntu, ACL, AzureLinux) correctly detect rcv1p mode
  • IsOptedInForRootCerts check works (skips on public cloud as expected)
  • Chrony configured per-OS, CSE completes successfully across 113 tests

Windows (Build 158446024):

  • Get-CACertificates -Location correctly selects rcv1p mode
  • Should-InstallCACertificatesRefreshTask returns $false on public cloud (correct)
  • Backward-compat guard works for older VHDs

TME tenant (RCV1P_SUBSCRIPTION_ID set in pipeline, with PlatformSettingsOverride feature flag)

Linux — Validated end-to-end: wireserver returns IsOptedInForRootCerts=true, certificates downloaded and installed into OS trust store, refresh schedule registered. Passed across Ubuntu 2204, Ubuntu 2404, AzureLinux V3, Flatcar, ACL.

Windows (Build 161633049):

  • 5 of 6 jobs passed — all RCV1P tests passed across Windows 2022, 23H2, 2025 (both gen1 and gen2 variants)
  • Wireserver returns IsOptedInForRootCerts=true, certificates downloaded to C:\ca, scheduled task aks-ca-certs-refresh-task registered
  • The only failure (windows-2022-containerd job) was a pre-existing Test_Windows2022_VHDCaching issue unrelated to RCV1P
  • Bug fix validated: The original Windows implementation used the wrong JSON field name (OperationRequests instead of OperationsInfo) when parsing wireserver responses — this was the root cause of empty cert downloads. Fixed in commit b6cd4e4f68.

Note on Windows E2E infrastructure: The published CSE scripts package (v0.0.52 from packages.aks.azure.com) predates the RCV1P code. To test the branch code end-to-end, the tests build a CSE zip from staging/cse/windows/ at test time, upload it to blob storage, and override CseScriptsPackageURL in the bootstrap config. This override is temporary — once a new CSE package is published with RCV1P support, the override can be removed.

Files Changed (31 files, +1979 / -1218)

Area Files Description
Linux provisioning init-aks-custom-cloud.sh, init-aks-custom-cloud-repos.sh (new), 3 removed Unified cert flow, repo depot extraction
Windows provisioning kubernetesfunc.ps1, kuberneteswindowssetup.ps1 rcv1p support, refresh task, backward compat
Windows tests kubernetesfunc.tests.ps1 (new) 260 lines of Pester tests
E2E tests scenario_rcv1p_test.go (new), scenario_rcv1p_win_test.go (new) Linux + Windows + negative tests
E2E infra vmss.go, types.go, validators.go, cluster.go, config/ Multi-sub, VM instance tags, validators
Pipeline e2e-rcv1p.yaml (new) Daily RCV1P test pipeline
AgentBaker service baker.go, const.go, variables.go Wire up new scripts

PR File Breakdown: Functionality vs Tests

Functionality (1,859 lines — 51%)

Lines File Purpose
455 parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Linux RCV1P cert provisioning
358 parts/linux/cloud-init/artifacts/init-aks-custom-cloud-repos.sh Split out repo init logic
346 parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh Deleted (consolidated)
236 parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh Deleted (consolidated)
211 staging/cse/windows/kubernetesfunc.ps1 Windows RCV1P cert functions
186 parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh Deleted (consolidated)
18 pkg/agent/variables.go Template variable plumbing
13 parts/windows/kuberneteswindowssetup.ps1 Windows CSE template
12 pkg/agent/const.go Embedded script references
7 parts/linux/cloud-init/nodecustomdata.yml Cloud-init custom data
7 aks-node-controller/parser/helper.go ANC parser helper
4 parts/linux/cloud-init/artifacts/cse_cmd.sh CSE command script
3 aks-node-controller/parser/templates/cse_cmd.sh.gtpl CSE command template
3 pkg/agent/baker.go Baker template plumbing

Tests / E2E Infra (1,795 lines — 49%)

Lines File Purpose
507 e2e/scenario_rcv1p_test.go Linux RCV1P E2E tests + CSE zip helper
260 staging/cse/windows/kubernetesfunc.tests.ps1 Windows PowerShell unit tests
195 e2e/scenario_rcv1p_win_test.go Windows RCV1P E2E tests
146 e2e/validators.go RCV1P validation functions
135 e2e/cluster.go Multi-subscription cluster support
111 e2e/vmss.go VMSS tagging and log collection
96 e2e/config/azure.go Azure client config for multi-sub
80 e2e/test_helpers.go Test helper improvements
68 e2e/types.go Scenario type extensions
61 spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh ShellSpec unit tests
47 e2e/cache.go Cluster cache additions
29 e2e/aks_model.go AKS model extensions
24 e2e/config/config.go E2E config additions
19 .pipelines/e2e-rcv1p.yaml Dedicated RCV1P pipeline
14 e2e/kube.go Kube helper improvements
2 .pipelines/scripts/e2e_run.sh E2E run script changes
1 .pipelines/templates/e2e-template.yaml E2E template changes

Summary

Category Lines Changed Percentage
Functionality 1,859 51%
Tests / E2E 1,795 49%
Total 3,654 100%

Copilot AI review requested due to automatic review settings March 16, 2026 05:14
Copilot AI reviewed Mar 16, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to unify the custom-cloud CA certificate bootstrap path (removing the separate “operation-requests” init scripts) and adds a Windows scheduled task to periodically refresh custom-cloud CA certificates.

Changes:

  • Windows: add a scheduled task to refresh custom-cloud CA certificates; update Get-CACertificates to support legacy vs “rcv1p” modes keyed off location.
  • Linux: consolidate custom-cloud init to a single init script and update CSE command generation to set a cert-endpoint mode variable.
  • Regenerate multiple custom data / generated command snapshots to reflect the new templates.

Reviewed changes

Copilot reviewed 74 out of 176 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
staging/cse/windows/kubernetesfunc.ps1 Adds CA refresh scheduled task + updates CA retrieval logic and error behavior
parts/windows/kuberneteswindowssetup.ps1 Wires Get-CACertificates -Location and registers refresh task for custom clouds
pkg/agent/variables.go Always injects initAKSCustomCloud payload into cloud-init data
pkg/agent/const.go Removes separate custom-cloud init script constants; keeps single init script
pkg/agent/baker.go Simplifies GetTargetEnvironment; notes IsAKSCustomCloud as deprecated
parts/linux/cloud-init/artifacts/cse_cmd.sh Updates CSE command to set cert endpoint mode + run custom-cloud init script
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh Deleted (custom-cloud init consolidation)
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh Deleted (custom-cloud init consolidation)
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh Deleted (custom-cloud init consolidation)
aks-node-controller/parser/templates/cse_cmd.sh.gtpl Mirrors CSE command template updates for aks-node-controller parser
aks-node-controller/parser/testdata/Compatibility+EmptyConfig/generatedCSECommand Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AzureLinuxv2+Kata+DisableUnattendedUpgrades=false/generatedCSECommand Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+SSHStatusOn/generatedCSECommand Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+EnablePubkeyAuth/generatedCSECommand New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+DisablePubkeyAuth/generatedCSECommand New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+DefaultPubkeyAuth/generatedCSECommand New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+CustomOSConfig/generatedCSECommand Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+CustomCloud/generatedCSECommand Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+Containerd+MIG/generatedCSECommand Regenerated snapshot for new CSE cmd template
aks-node-controller/parser/testdata/AKSUbuntu2204+CloudProviderOverrides/generatedCSECommand New snapshot for new template output
aks-node-controller/parser/testdata/AKSUbuntu2204+China/generatedCSECommand Regenerated snapshot for new CSE cmd template
pkg/agent/testdata/MarinerV2+Kata/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+China/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOff/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/CustomizedImage/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/CustomizedImageKata/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData Regenerated snapshot (custom data gzip payload changed)
pkg/agent/testdata/Flatcar/CustomData.inner Regenerated snapshot (embedded gzip payload changed)
pkg/agent/testdata/ACL/CustomData.inner Regenerated snapshot (embedded gzip payload changed)

You can also share your feedback on Copilot code review. Take the survey.

Comment thread staging/cse/windows/kubernetesfunc.ps1
Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated
Comment thread parts/linux/cloud-init/artifacts/cse_cmd.sh Outdated
Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl Outdated
@rchincha rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from 44ff9ee to a0a1307 Compare March 18, 2026 21:11
Copilot AI review requested due to automatic review settings March 18, 2026 22:12
Copilot AI reviewed Mar 18, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to unify AKS custom-cloud CA certificate bootstrap behavior (legacy vs “rcv1p/operation-requests” style flows) and adds a Windows scheduled task to periodically refresh custom-cloud CA certificates.

Changes:

  • Adds Windows CA refresh scheduled task registration and introduces location-based endpoint-mode selection (legacy vs rcv1p).
  • Refactors Windows CA certificate retrieval to support both endpoint modes and opt-in gating for rcv1p.
  • Simplifies Linux custom-cloud init script selection by consolidating onto init-aks-custom-cloud.sh and removing older variants; updates generated testdata accordingly.

Reviewed changes

Copilot reviewed 93 out of 99 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
staging/cse/windows/kubernetesfunc.ps1 Adds CA refresh scheduled task and endpoint-mode-aware Get-CACertificates implementation.
pkg/agent/variables.go Simplifies how initAKSCustomCloud is added to Linux cloud-init variables.
pkg/agent/testdata/MarinerV2+Kata/CustomData Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/Flatcar/CustomData.inner Updates expected Flatcar CustomData snapshot (generated content changed).
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/CustomizedImageKata/CustomData Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/CustomizedImage/CustomData Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData Updates expected CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S119/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S118/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S117/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+K8S116/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData Updates expected Windows CustomData snapshot (calls/refresh task additions).
pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData Updates expected Windows CustomData snapshot (new Get-CACertificates call form + refresh task).
pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData Updates expected Windows CustomData snapshot (new Get-CACertificates call form + refresh task).
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/AKSUbuntu2204+China/CustomData Updates expected Ubuntu CustomData snapshot (generated content changed).
pkg/agent/testdata/ACL/CustomData.inner Updates expected ACL CustomData snapshot (generated content changed).
pkg/agent/const.go Consolidates custom-cloud init script constants to a single script.
parts/windows/kuberneteswindowssetup.ps1 Updates Windows setup flow to call Get-CACertificates with location and registers CA refresh scheduled task.
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh Removes operation-requests-specific Linux init script (consolidation).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh Removes Mariner/AzureLinux operation-requests init script (consolidation).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh Removes Mariner/AzureLinux legacy init script variant (consolidation).
aks-node-controller/parser/templates/cse_cmd.sh.gtpl Adds a LOCATION shell variable in the generated CSE command template.
aks-node-controller/parser/helper.go Factors out a shared getCloudLocation helper and reuses it in getCloudTargetEnv.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated
Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated
Comment thread staging/cse/windows/kubernetesfunc.ps1
Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl Outdated
@rchincha rchincha mentioned this pull request Mar 19, 2026
Closed
@rchincha rchincha changed the title feat(custom-cloud): unify cert bootstrap flow and add Windows CA refresh task feat(rcv1p): unify cert bootstrap flow and add Windows CA refresh task Mar 19, 2026
@rchincha rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from 2b3c1d6 to e19a19b Compare March 19, 2026 00:51
Copilot AI review requested due to automatic review settings March 19, 2026 01:00
@rchincha rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from e19a19b to d41856f Compare March 19, 2026 01:00
Copilot AI reviewed Mar 19, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR unifies the AKS custom cloud CA certificate bootstrap logic to a single flow and adds a Windows scheduled task to periodically refresh custom cloud CA certificates. It also updates Linux/customdata generation and test snapshots to reflect the new wiring.

Changes:

  • Add Windows scheduled task registration for daily CA certificate refresh and introduce a location-based cert endpoint mode selector.
  • Simplify Linux custom cloud init script selection by standardizing on init-aks-custom-cloud.sh, plus add wiring/tests for refresh-mode arguments.
  • Update aks-node-controller template to export LOCATION, and regenerate CustomData snapshot test artifacts.

Reviewed changes

Copilot reviewed 95 out of 101 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
staging/cse/windows/kubernetesfunc.tests.ps1 Adds Pester coverage for cert endpoint mode selection, scheduled task registration, and CA retrieval behavior.
staging/cse/windows/kubernetesfunc.ps1 Implements unified Windows CA retrieval logic with legacy/rcv1p modes and registers a daily refresh scheduled task.
spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh Adds ShellSpec assertions to validate refresh-mode argument parsing/wiring in the Linux init script.
pkg/agent/variables.go Changes how initAKSCustomCloud is injected into Linux cloud-init data.
pkg/agent/const.go Removes per-cloud custom init script constants and standardizes on init-aks-custom-cloud.sh.
parts/windows/kuberneteswindowssetup.ps1 Wires CA retrieval call and registers the Windows CA refresh scheduled task during BasePrep.
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh Removed (operation-requests variant no longer used).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh Removed (operation-requests Mariner/AzureLinux variant no longer used).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh Removed (Mariner/AzureLinux legacy variant no longer used).
aks-node-controller/parser/templates/cse_cmd.sh.gtpl Exports LOCATION into the CSE environment for downstream scripts.
aks-node-controller/parser/helper.go Adds a helper to normalize location and reuses it in cloud target env detection.
pkg/agent/testdata/MarinerV2+Kata/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/Flatcar/CustomData.inner Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/CustomizedImageKata/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/CustomizedImage/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S119/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S118/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S117/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+K8S116/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData Regenerated CustomData snapshot due to Windows CA refresh task wiring.
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/AKSUbuntu2204+China/CustomData Regenerated CustomData snapshot due to init/custom cloud wiring changes.
pkg/agent/testdata/ACL/CustomData.inner Regenerated CustomData snapshot due to init/custom cloud wiring changes.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread parts/windows/kuberneteswindowssetup.ps1 Outdated
Comment thread pkg/agent/variables.go
@rchincha rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from d41856f to 18ba549 Compare March 19, 2026 20:13
Copilot AI review requested due to automatic review settings March 19, 2026 22:07
@rchincha rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from 18ba549 to e94c465 Compare March 19, 2026 22:07
Copilot AI reviewed Mar 19, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates AKS custom cloud certificate bootstrapping to use a single unified flow and adds a Windows scheduled task for periodic custom cloud CA refresh.

Changes:

  • Added Windows CA refresh task registration plus new logic to select cert retrieval mode and opt-in gating.
  • Simplified Linux custom cloud init script wiring by removing legacy “operation-requests” variants and normalizing location for refresh mode.
  • Added/updated tests and refreshed golden testdata outputs to reflect new custom data content.

Reviewed changes

Copilot reviewed 95 out of 101 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
staging/cse/windows/kubernetesfunc.tests.ps1 Adds Pester coverage for endpoint-mode selection, task registration behavior, and CA retrieval failure handling.
staging/cse/windows/kubernetesfunc.ps1 Implements endpoint-mode derivation, opt-in gating, CA retrieval paths, and a Windows scheduled task for refresh.
spec/parts/linux/cloud-init/artifacts/init_aks_custom_cloud_spec.sh Adds ShellSpec checks to ensure init script wiring for ca-refresh mode and LOCATION usage.
pkg/agent/variables.go Simplifies init script selection and updates how custom cloud init script is injected into cloud-init data.
pkg/agent/const.go Removes now-unused custom-cloud init script constants; keeps unified init script constant.
parts/windows/kuberneteswindowssetup.ps1 Updates Windows setup to call Get-CACertificates with Location and conditionally register refresh task.
aks-node-controller/parser/templates/cse_cmd.sh.gtpl Adds LOCATION variable for downstream scripts during custom cloud provisioning.
aks-node-controller/parser/helper.go Adds getCloudLocation helper and reuses it for cloud target env detection.
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests.sh Removes legacy operation-requests init script (superseded by unified script).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-operation-requests-mariner.sh Removes legacy Mariner operation-requests init script (superseded by unified script).
parts/linux/cloud-init/artifacts/init-aks-custom-cloud-mariner.sh Removes legacy Mariner init script variant (superseded by unified script).
pkg/agent/testdata/MarinerV2+Kata/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/Flatcar/CustomData.inner Updates golden ignition/customData payload for unified custom cloud init content.
pkg/agent/testdata/CustomizedImageLinuxGuard/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/CustomizedImageKata/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/CustomizedImage/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AzureLinuxV3+Kata/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AzureLinuxV2+Kata/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingNoConfig/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworkingDisabled/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows23H2Gen2+NextGenNetworking/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+ootcredentialprovider/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+SecurityProfile/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+ManagedIdentity/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+KubeletServingCertificateRotation/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+KubeletClientTLSBootstrapping/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S119/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S119+FIPS/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S119+CSI/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S118/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S117/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+K8S116/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+EnablePrivateClusterHostsConfigAgent/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+CustomVnet/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+CustomCloud/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSWindows2019+CustomCloud+ootcredentialprovider/CustomData Updates golden Windows customData to pass Location to CA cert retrieval + refresh task gating.
pkg/agent/testdata/AKSUbuntu2404+Teleport/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2404+NetworkPolicy/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+ootcredentialprovider/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+cgroupv2/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+SecurityProfile/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOn/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+SSHStatusOff/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/AKSUbuntu2204+China/CustomData Updates golden customData to match unified custom cloud init content.
pkg/agent/testdata/ACL/CustomData.inner Updates golden ignition/customData payload for unified custom cloud init content.
Comments suppressed due to low confidence (7)

staging/cse/windows/kubernetesfunc.ps1:1

  • Get-CACertificates used to fail fast via Set-ExitCode on retrieval/parse errors, but now returns $false (and logs warnings) for a wide range of failure cases. Because call sites in the generated setup scripts invoke Get-CACertificates -Location $Location without checking the return value, this can silently proceed without required CA material and lead to harder-to-diagnose TLS failures later in provisioning. Consider restoring fatal behavior for “expected-to-install” scenarios (e.g., legacy mode, or rcv1p when opted-in), or have callers check the return value and invoke Set-ExitCode when it’s $false in those modes.
    staging/cse/windows/kubernetesfunc.ps1:1
  • Get-CACertificates used to fail fast via Set-ExitCode on retrieval/parse errors, but now returns $false (and logs warnings) for a wide range of failure cases. Because call sites in the generated setup scripts invoke Get-CACertificates -Location $Location without checking the return value, this can silently proceed without required CA material and lead to harder-to-diagnose TLS failures later in provisioning. Consider restoring fatal behavior for “expected-to-install” scenarios (e.g., legacy mode, or rcv1p when opted-in), or have callers check the return value and invoke Set-ExitCode when it’s $false in those modes.
    pkg/agent/variables.go:1
  • This change removes the previous cs.IsAKSCustomCloud() guard and injects the custom cloud init script into cloudInitData unconditionally. That can increase customData size for all clusters (risking platform limits) and may introduce unintended side effects if any downstream template writes/executes this script outside custom cloud. Recommend reinstating the custom cloud guard (and only setting initAKSCustomCloud when IsAKSCustomCloud() is true), while still using the unified initAKSCustomCloudScript for all custom clouds.
    staging/cse/windows/kubernetesfunc.ps1:1
  • $resourceFileName is used directly to build a path under C:\ca. If the upstream response ever contains path separators (e.g., ..\foo or nested paths), this can write outside the intended directory. Prefer sanitizing to a basename (e.g., using Split-Path -Leaf or [IO.Path]::GetFileName($resourceFileName)) before Join-Path, and consider rejecting names containing directory traversal characters.
    staging/cse/windows/kubernetesfunc.ps1:1
  • $resourceFileName is used directly to build a path under C:\ca. If the upstream response ever contains path separators (e.g., ..\foo or nested paths), this can write outside the intended directory. Prefer sanitizing to a basename (e.g., using Split-Path -Leaf or [IO.Path]::GetFileName($resourceFileName)) before Join-Path, and consider rejecting names containing directory traversal characters.
    staging/cse/windows/kubernetesfunc.ps1:1
  • The new rcv1p operation-requests flow is non-trivial (multiple requests, JSON shape assumptions, per-item content downloads, and $downloadedAny aggregation), but the added Pester tests only cover legacy mode and the “throws returns false” path. Add tests that (1) exercise the rcv1p path end-to-end with mocked Retry-Command returning operation requests and cert bodies, and (2) verify behavior when operation requests are empty/invalid (ensuring the function returns $false and logs expected warnings).
    pkg/agent/variables.go:1
  • The PR description still contains placeholder text (Fixes # with no linked issue and no explanation of “what/why”). Please update the PR description to summarize the behavior change (unified bootstrap + Windows refresh task) and link the relevant issue or remove the placeholder.

Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl
@rchincha rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from e94c465 to f20d5b8 Compare March 19, 2026 23:28
Copilot AI review requested due to automatic review settings March 20, 2026 06:42
@rchincha rchincha force-pushed the origin/rchinchani/rcv1p-2 branch from f20d5b8 to b53f240 Compare March 20, 2026 06:42
rchincha and others added 30 commits May 20, 2026 11:02
@rchincha
REVERT ME: add wireserver endpoint diagnostics to Windows RCV1P valid…
4d5ca4e 
…ator

Probes all wireserver cert endpoints (isOptedInForRootCerts,
operationrequestsroot, operationrequestsintermediate, legacy
cacertificates) during validation and dumps CSE log lines related
to certificate operations. Uses execScriptOnVMForScenario with
explicit t.Logf to ensure output is always visible in test logs,
not swallowed by execScriptOnVMForScenarioValidateExitCode.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: use correct wireserver JSON field name for rcv1p cert download
b936a2f 
The wireserver operationrequestsroot and operationrequestsintermediate
endpoints return certificates under the 'OperationsInfo' field, but
the Windows PowerShell code was looking for 'OperationRequests' which
doesn't exist in the response. This caused the null check to skip the
entire cert download loop, leaving C:\ca empty despite wireserver
returning valid certificate data.

The Linux implementation avoids this by using grep to extract
ResouceFileName values directly from the raw JSON, bypassing the
parent field name entirely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
REVERT ME: add azcopy error logging to Windows log collection
15c0077 
Wraps each azcopy copy call with error checking and logging to
diagnose why Windows CSE log uploads consistently return BlobNotFound.
Also captures RunCommand stdout/stderr (InstanceView) which was
previously not logged, so we can see azcopy output and any MSI
auth failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
REVERT ME: enable verbose test output for azcopy/wireserver diagnostics
2efa0f2 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
REVERT ME: canary check to prove whether SSH validators are broken
ce1a29f 
Adds a ValidateFileHasContent check for a nonsense string that will
never exist in the CSE log. If this test PASSES, it proves the
ExitMissingError handler in exec.go:130 is silently swallowing
SSH exit codes and all Windows validators are no-ops.
If this test FAILS (expected), validators are working correctly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
Remove canary check - validators confirmed working
fcc42a9 
The canary test proved validators are functional and our branch CSE
zip is correctly delivered to VMs. Wireserver returns
IsOptedInForRootCerts=true and the CSE log contains the expected
RCV1P log lines.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: make wireserver cert retrieval failures fatal on Linux
d37eeae 
Cert installation must succeed for the selected mode (legacy or rcv1p).
Previously, failures after exhausting retries were silently swallowed
with a warning, leaving the node without certificates. Now failures
exit 1, matching the Windows -FailOnError behavior.

Retries with backoff in make_request_with_retry still handle transient
wireserver issues (rate limiting, temporary unavailability).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
revert: remove diagnostic commits used during RCV1P development
5ff8a25 
Reverts the following temporary diagnostic commits that served their
purpose during RCV1P cert mode debugging and are no longer needed:

- 807b5a4 (wireserver endpoint diagnostics in validator)
  Why: Added to debug cert download failures. The root cause was a
  JSON field name mismatch (OperationRequests vs OperationsInfo),
  now fixed. Diagnostic probing adds noise to validator output.

- 9f6a902 (azcopy error logging in Windows log collection)
  Why: Added to debug empty CSE log uploads (BlobNotFound). Root
  cause was ADO job timeout (90m) racing with go test timeout (90m),
  fixed on main by 54aa84a (reduced go test timeout to 80m).

- d083fbe (verbose test output with -v flag)
  Why: Added so t.Logf output would appear in pipeline logs for
  diagnostics. No longer needed; increases log noise for all tests.

- 45041cb (always collect Windows CSE logs)
  Why: Removed s.T.Failed() guard to collect logs on success too.
  Root cause of missing logs was the ADO/go-test timeout race,
  not the collection logic. Restored failure-only collection.

- fdc6962 + 1196773 (canary check, already net-zero)
  Why: Canary proved validators work correctly. Already removed
  by the follow-up commit; these two commits cancel each other.

- 0bc8f2e (poll wireserver IsOptedInForRootCerts retry loop)
  Why: Experimental polling for FC goal-state propagation. Tags
  are now set at VMSS creation time, making polling unnecessary.
  Already reverted by later commits during development.

Kept (not reverted):
- 76edb18: Azure CNI cluster for Windows RCV1P tests (real fix
  for NBC/cluster type mismatch causing IP exhaustion)
- a891055: Branch-built CSE zip override (required until RCV1P
  code ships in a published CSE package)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: make wireserver unreachable fatal for RCV1P opt-in check
dfb2c10 
Wireserver unreachable after retries is now fatal (return 2 + exit 1)
instead of silently skipping cert installation. If the subscription is
opted in for hardened root certs but we silently fall back to the
distro's default trust store, we leave a security hole — the node
would trust CAs the customer explicitly intended to replace.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: use RCV1P Azure CNI cluster for Windows tests when explicit subs…
8c89063 
…cription set

When RCV1P_SUBSCRIPTION_ID is set, Windows RCV1P positive tests set
Scenario.AzureClient/SubscriptionID to the RCV1P subscription, but
rcv1pWindowsCluster() always returned ClusterAzureNetwork (default
subscription). This subscription mismatch would cause VMSS creation
to 404 in the RCV1P subscription's node resource group.

Fix:
- Add ClusterRCV1PAzureNetwork in cache.go (Azure CNI cluster using
  RCV1PClusterInfra)
- Branch rcv1pWindowsCluster() on hasExplicitRCV1PSubscription(),
  matching the pattern used by rcv1pCluster() for Linux
- Fix Test_RCV1P_Windows_NotOptedIn to use ClusterRCV1PAzureNetwork
  instead of ClusterRCV1PKubenet (Windows needs Azure CNI)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: replace legacy ca-refresh cron entry with location-aware version
4ada2fe 
On custom clouds (AGC, Delos) where an older version of this script
already installed a ca-refresh cron entry without the location argument,
the idempotency grep would match the old entry and skip adding the new
one. The old cron entry runs ca-refresh with an empty location, causing
get_cert_endpoint_mode to default to rcv1p instead of legacy for
ussec/usnat environments.

Fix: always remove any existing ca-refresh entry for this script and
re-add it with the explicit location argument, ensuring upgraded nodes
get the correct endpoint mode on periodic refresh.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: align Windows wireserver retries to 10 to match Linux parity
cf07a71 
All wireserver Retry-Command calls in kubernetesfunc.ps1 increased
from 5 to 10 retries, matching Linux make_request_with_retry which
uses 10 retries with exponential backoff. Under rate-limiting or
transient wireserver unavailability, 5 retries (50s) could exhaust
before the endpoint recovers.

Added comments explaining:
- Retry count parity with Linux
- Security rationale: wireserver unreachable with -FailOnError is
  fatal because silently falling back to the OS default trust store
  would be a security hole if the customer intended hardened certs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: enhance RCV1P opt-in tag handling in VMSS creation process
09b8d20 
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: use Azure CNI cluster for Windows RCV1P tests
b99bede 
Windows baseTemplateWindows() configures NBC with NetworkPlugin=azure and
NetworkPluginMode=overlay. Using a kubenet cluster causes azure-vnet plugin
IPAM failures on the node. Switch all Windows RCV1P tests to use
ClusterRCV1PAzureNetwork which creates an Azure CNI overlay cluster in the
RCV1P subscription.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
revert: drop 'REVERT ME' cluster switching commits (now superseded)
cafa6ec 
The following commits are superseded by the permanent fix in c71b1eb
which correctly assigns ClusterRCV1PAzureNetwork to Windows RCV1P tests
and keeps ClusterRCV1PKubenet for Linux RCV1P tests:

- 286c711 REVERT ME: use dedicated kubenet cluster for RCV1P tests
- 4de7fe5 REVERT ME: use Azure CNI cluster for Windows RCV1P tests

Both are no-ops against the current state and can be safely squashed out
during final interactive rebase before merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
revert: drop canary validator and wireserver polling debug commits
00804a3 
Reverts:
- 5c2ed65 (canary check that guarantees test failure)
- 07d1c44 (5-minute wireserver polling loop - provisioning regression)

The canary ValidateFileHasContent for a nonexistent string causes
guaranteed test failures. The wireserver polling adds up to 5 minutes
of sleep to every Linux RCV1P node provisioning.

Remaining diagnostic commits (wireserver endpoint probing, azcopy
logging, verbose output) are kept for initial rollout observability.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
feat(e2e): auto-detect RCV1P feature flag on E2E subscription
5904637 
When RCV1P_SUBSCRIPTION_ID is not explicitly set, the skip logic now
checks whether the E2E subscription (E2E_SUBSCRIPTION_ID) has the
PlatformSettingsOverride feature flag registered. If it does, the
RCV1P tests run automatically using the E2E subscription.

This enables MSFT tenant pipelines (where the E2E subscription is
already enrolled) to run RCV1P tests without a separate variable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix(e2e): skip NotOptedIn tests on auto-detected enrolled subscriptions
c709eae 
On subscriptions with PlatformSettingsOverride registered, the platform
auto-injects the opt-in tag on ALL VMSSes, making the 'not opted in'
negative test scenario impossible. Skip these tests when the RCV1P
subscription was auto-detected from the E2E subscription.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix(e2e): use caller context in getCustomScriptExtensionStatus
e1959ba 
Replace context.Background() with the caller's context so the VM
instance view fetch respects test/scenario timeouts instead of
potentially hanging indefinitely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix(e2e): remove TrustedLaunch from non-Gen2 Windows 2025 RCV1P test
dee66b3 
The windows-2025 image does not support TrustedLaunch, only the Gen2
variant does. This matches the pattern on main where Test_Windows2025
uses EmptyVMConfigMutator without TrustedLaunch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: return code 2 when wireserver is unreachable in is_opted_in_for_…
117adaa 
…root_certs

The function documented return code 2 for 'wireserver unreachable' and
the caller correctly checked for it, but the implementation returned 1
(not opted in) on request failure. This silently skipped cert installation
on wireserver outages — a security hole if the subscription is enrolled
for hardened certs. Now returns 2 on failure so the caller treats it as
fatal, matching the documented contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: throw when opted-in but no certs downloaded with -FailOnError
2cb6a99 
When IsOptedInForRootCerts is true but no certificates are downloaded,
Get-CACertificates only logged a warning and returned \False. Because
the caller (BasePrep) doesn't check the return value, provisioning
continued without the required CA set. Now throws when -FailOnError is
set and no certs were downloaded, matching the fail-closed contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
e2e: use branch-built CSE zip for Windows RCV1P tests
ccc57f8 
The published CSE package (aks-windows-cse-scripts-current.zip) does not
contain the RCV1P code (Get-CACertificates -Location, -FailOnError,
IsOptedInForRootCerts, Register-CACertificatesRefreshTask). Without this
override, Windows RCV1P E2E tests pass vacuously using the old code path.

This builds a CSE zip from staging/cse/windows/ at test time, uploads it
to blob storage with a SAS URL, and overrides CseScriptsPackageURL so
the VMs download the branch's CSE scripts.

TODO(rcv1p): remove the branch CSE zip override and rcv1pWindowsCSEMutator
once the RCV1P code ships in a published CSE package.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: parse wireserver IsOptedInForRootCerts JSON response with jq
1efb2c0 
The wireserver returns JSON like {"IsOptedInForRootCerts":true} but
the script was using grep for IsOptedInForRootCerts=true (equals
sign), which never matches the JSON colon format. Use jq for proper
JSON parsing instead.

This fix was previously applied but accidentally dropped during a
rebase squash/reorder.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix(e2e): update BootstrapConfigMutator signatures after rebase
d1414df 
Adapt to upstream signature change: BootstrapConfigMutator now takes
(*Cluster, *NodeBootstrappingConfiguration) instead of just
(*NodeBootstrappingConfiguration). Also thread infra parameter through
setupPrivateDNSForAPIServer to match getClusterVNet signature.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: fail process_cert_operations when no cert bodies are saved
224576e 
Track the number of successfully saved certificates and return non-zero
if all individual cert content fetches failed despite the operation
endpoint returning filenames. This closes a gap where retrieve_rcv1p_certs
could report success with zero certs actually downloaded.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: pass repodepot_endpoint explicitly to add_key_ubuntu and add_ms_…
f757ce8 
…keys

These functions relied on bash dynamic scoping to access the caller's
local repodepot_endpoint variable. Pass it as an explicit parameter
to follow the repo's shell script guidelines and avoid fragile implicit
variable dependencies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
chore(e2e): remove REVERT ME wireserver diagnostic block from Windows…
796d1ff 
… validator

Remove the always-on diagnostic block that probed wireserver endpoints
and dumped CSE logs on every Windows RCV1P test run. This bloated test
logs, added latency, and could leak wireserver response content into CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix(e2e): use infra.Azure for private DNS operations in RCV1P subscri…
482ec2d 
…ption

createPrivateZone, waitForPrivateZone, createPrivateDNSLink, and the
RecordSet call in setupPrivateDNSForAPIServer were hardcoded to
config.Azure (the default E2E subscription). When running RCV1P tests
in a separate subscription, the MC_ resource group only exists in the
RCV1P subscription, causing ResourceGroupNotFound errors.

Add an azure *config.AzureClient parameter to these functions so the
caller can pass the correct subscription client. setupPrivateDNSForAPIServer
now uses infra.Azure; addPrivateEndpointForACR continues using config.Azure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
@rchincha
fix: guard against unresolved ADO pipeline variable expressions in RC…
3c0baac 
…V1PSubscriptionID

Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@r2k1 r2k1 r2k1 left review comments

@cameronmeissner cameronmeissner cameronmeissner left review comments

Copilot code review Copilot Copilot left review comments

@ganeshkumarashok ganeshkumarashok Awaiting requested review from ganeshkumarashok ganeshkumarashok is a code owner

@Devinwong Devinwong Awaiting requested review from Devinwong Devinwong is a code owner

@lilypan26 lilypan26 Awaiting requested review from lilypan26 lilypan26 is a code owner

@AbelHu AbelHu Awaiting requested review from AbelHu AbelHu is a code owner

@junjiezhang1997 junjiezhang1997 Awaiting requested review from junjiezhang1997 junjiezhang1997 is a code owner

@djsly djsly Awaiting requested review from djsly djsly is a code owner

@phealy phealy Awaiting requested review from phealy phealy is a code owner

@timmy-wright timmy-wright Awaiting requested review from timmy-wright timmy-wright is a code owner

@zachary-bailey zachary-bailey Awaiting requested review from zachary-bailey zachary-bailey is a code owner

@awesomenix awesomenix Awaiting requested review from awesomenix awesomenix is a code owner

@mxj220 mxj220 Awaiting requested review from mxj220 mxj220 is a code owner

@pdamianov-dev pdamianov-dev Awaiting requested review from pdamianov-dev pdamianov-dev is a code owner

@calvin197 calvin197 Awaiting requested review from calvin197 calvin197 is a code owner

@sulixu sulixu Awaiting requested review from sulixu sulixu is a code owner

@surajssd surajssd Awaiting requested review from surajssd surajssd is a code owner

@SriHarsha001 SriHarsha001 Awaiting requested review from SriHarsha001 SriHarsha001 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rchincha @r2k1 @cameronmeissner