Skip to content

fix AKS node counting, enumerate VMSS-backed pools accurately, add Container Apps replica counting, update ResourceManagementClient import for azure-mgmt-resource 26.0.0; also let benchmark.sh use the local Azure script - #75

Merged
ryanjpayne merged 5 commits into
CrowdStrike:mainfrom
jzcwong:azurefix
Jul 28, 2026

Conversation

@jzcwong

@jzcwong jzcwong commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Five related changes — four to Azure/azure_cspm_benchmark.py (plus its requirements.txt) and one to benchmark.sh:

  1. Update ResourceManagementClient import path for azure-mgmt-resource 26.0.0.
  2. Fix a TypeError crash when an AKS agent pool reports count = None (autoscaler-managed or virtual-node pools).
  3. Replace the AgentPool.count lookup with direct VMSS-instance enumeration for accurate node counts across all pool types.
  4. Add a new container_app_replicas column that counts only actually-running replicas across Azure Container Apps in each subscription.
  5. benchmark.sh: mirror the AWS branch behavior for Azure — use ../Azure/azure_cspm_benchmark.py when present, otherwise fall back to the remote download.

Files changed

  • Azure/azure_cspm_benchmark.py
  • Azure/requirements.txt
  • benchmark.sh

1. Import fix: ResourceManagementClient for azure-mgmt-resource 26.0.0

Azure/azure_cspm_benchmark.py:20

-from azure.mgmt.resource import ResourceManagementClient
+from azure.mgmt.resource.resources import ResourceManagementClient

Why: Update Azure code as per release 26.0.0 of azure-mgmt-resource, same rationale as PR #74. The top-level azure.mgmt.resource re-export of
ResourceManagementClient was removed; consumers must import it from the .resources submodule.


2. Fix: AKS node-pool crash on None node count

Previously the script crashed with:

TypeError: %d format: a real number is required, not NoneType
...
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'

whenever an AKS AgentPool returned count = None. This happens for pools where the cluster autoscaler owns scaling and for virtual-node pools — the Azure API leaves count unset in those cases.

Superseded by change (3) below, which stops relying on AgentPool.count entirely.


3. Accurate AKS node counting via VMSS enumeration

Rewrote AzureHandle.container_vmss(aks_resource). It previously listed agent pools via container_client.agent_pools.list(...) and returned AgentPool objects. It now:

  • Fetches the cluster via managed_clusters.get(...) to read node_resource_group (the auto-generated MC_<rg>_<cluster>_<region> group).
  • Lists every VMSS in that node RG via virtual_machine_scale_sets.list(...).
  • Counts VM instances per VMSS via virtual_machine_scale_set_vms.list(...).
  • Yields (pool_name, vm_count) tuples. Pool name is resolved from the aks-managed-poolName tag (fallback: poolName tag, then VMSS name).

Also:

  • Added AzureHandle.compute_client(subscription_id) — an lru_cached ComputeManagementClient factory.
  • Refactored vms_inside_vmss to use it instead of instantiating a fresh client per call.
  • Updated the call site in main() step (1) to unpack the new tuple shape.

Trade-offs

  • More API calls per cluster than the old single agent_pools.list (one managed_clusters.get + one VMSS list + one VM list per VMSS). Chattier but accurate.
  • Virtual-node (ACI-backed) pools have no VMSS and will contribute 0 to aks_nodes. This matches billing reality (virtual nodes are not VMs); if a future PR wants to surface them, they should be folded into
    aci_containers or a separate column.
  • Legacy VMAS-based pools (deprecated for new AKS clusters) are not enumerated. Unlikely to affect anyone in practice.
  • The calling identity needs Microsoft.Compute/virtualMachineScaleSets/*/read on the AKS node resource group (the built-in Reader role covers this).

4. New column: Azure Container Apps running-replica count

New dependency

Added azure-mgmt-appcontainers to Azure/requirements.txt.

New import

from azure.mgmt.appcontainers import ContainerAppsAPIClient

New column

container_app_replicas ("Container App Replicas") added to headers, totals, and each per-subscription row. Appears in both the terminal tabulate output and the azure-benchmark.csv file.

New AzureHandle methods

Method Purpose
container_apps(subscription_id) Lists every Container App in the subscription via container_apps.list_by_subscription().
container_app_running_replicas(container_app) For each active revision of the app, enumerates the container_apps_revision_replicas subresource and counts only replicas whose running_state == 'Running'.
container_apps_client(subscription_id) lru_cached ContainerAppsAPIClient factory.

Wired into main() as step (5). Wrapped in try/except so subscriptions without the Microsoft.App resource provider registered log a warning and continue rather than fail the whole run. Per-revision replica-list calls
are individually try/excepted so throttling or transient 5xx errors on one revision only cost that revision, not the whole run.

Semantics

  • Only replicas the platform reports as running_state == 'Running' are counted.
  • Replicas the portal shows as Provisioning, Failed, Degraded, Stopped, or Unknown are excluded.
  • Inactive revisions (revision.active == False) are skipped entirely.

Naïvely summing revision.replicas (which is what an earlier draft of this PR did) over-counts live workload because that field is a scheduling count, not a runtime-health count.

Caveats

  • API-call cost. Roughly 2–3× the ARM calls of a revision.replicas-summing approach for this section: 1 list-revisions per app + 1 list-replicas per active revision. Container Apps ARM has aggressive throttling on
    large tenants; the per-revision try/except keeps the run alive under partial failure but you may see under-counts logged as warnings when throttled.
  • Live gauge. The result is a snapshot. Autoscaling apps and apps mid-cold-start will legitimately differ between back-to-back runs.
  • Scale-to-zero apps report 0 even though they'd bill compute when invoked. If provisioned-capacity semantics matter, a future PR could switch to summing template.scale.minReplicas per active revision instead.
  • Microsoft.App/jobs are a separate resource type and are intentionally not counted here.
  • Required RBAC: Microsoft.App/containerApps/read, Microsoft.App/containerApps/revisions/read, Microsoft.App/containerApps/revisions/replicas/read (the built-in Reader role covers all three).

5. benchmark.sh: use local Azure script when present

benchmark.sh:151-172 — the Azure|GCP) case now checks for ../Azure/azure_cspm_benchmark.py (relative to the ./cloud-benchmark/ venv, i.e., the repo root) before falling back to curling from the pinned upstream
release tag. When the local file exists, benchmark.sh uses it plus ../Azure/requirements.txt, falling back to remote requirements only if the local requirements.txt is missing. Mirrors the AWS branch pattern exactly.

Why: Previously, benchmark.sh azure ignored any local edits in Azure/ and always downloaded azure_cspm_benchmark.py from
https://raw.githubusercontent.com/CrowdStrike/cloud-resource-estimator/${RELEASE_VERSION}/Azure/... (currently pinned to v1.0.0). That made it impossible to test local Azure changes end-to-end through the same entry
point most users invoke.

Scope: Only the Azure branch is affected. GCP still always downloads from remote — the Azure|GCP) case stays joint, but the new local-file check only fires when $CLOUD == "Azure". Extending it to GCP is a two-line
follow-up if maintainers want it.


Not changed (worth calling out)

  • Powered-off / deallocated Azure VMs are still counted in the vms column. Unchanged from prior behavior and out of scope for this PR — happy to open a follow-up if maintainers want to align with the AWS script's behavior
    of separating running from non-running.
  • The AKS-managed VMSS skip check in step (2) of main() still keys on the aks-managed-createOperationID tag. Not changed here — no evidence yet of it missing AKS scale sets in the wild.
  • The AWS and new Azure branches of benchmark.sh both use a hard-coded ../AWS/... / ../Azure/... relative path that only resolves because benchmark.sh does pushd ./cloud-benchmark before calling audit(). A future
    refactor of that pushd would silently break both. A code comment explaining the relative-path assumption would be a worthwhile follow-up, kept out of scope here.

Testing

  • Ran Azure/azure_cspm_benchmark.py directly against a subscription with an AKS cluster containing an autoscaler-managed pool (workerpool1) that previously triggered the NoneType crash — now completes cleanly and
    reports the correct instance count.
  • Container App replica count cross-checked against the portal's Revisions and replicas blade — script's total matches the count of replicas whose portal status is Running, and excludes replicas in
    Provisioning/Failed/Degraded/Stopped/Unknown.
  • benchmark.sh azure end-to-end verification is straightforward once merged: from the repo root, ./benchmark.sh azure should print Using local Azure CSPM benchmark script... before proceeding.

jzcwong added 5 commits July 27, 2026 15:13
Mirror the AWS branch behavior for Azure so that a local
../Azure/azure_cspm_benchmark.py (and requirements.txt) is used when
present, instead of always downloading from the pinned upstream
release tag. Falls back to the remote download when the local file
is missing.
Previously container_app_running_replicas summed revision.replicas across
active revisions, which includes replicas the portal shows as
Provisioning, Failed, Degraded, Stopped, or Unknown — over-counting live
workload. Enumerate the container_apps_revision_replicas subresource
per active revision and count only replicas whose running_state is
'Running'. Failed replica-list calls log a warning and skip that
revision rather than aborting the run.
@ryanjpayne ryanjpayne self-assigned this Jul 28, 2026
@ryanjpayne
ryanjpayne merged commit 38e7160 into CrowdStrike:main Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants