fix AKS node counting, enumerate VMSS-backed pools accurately, add Container Apps replica counting, update ResourceManagementClient import for azure-mgmt-resource 26.0.0; also let benchmark.sh use the local Azure script - #75
Merged
Conversation
Mirror the AWS branch behavior for Azure so that a local ../Azure/azure_cspm_benchmark.py (and requirements.txt) is used when present, instead of always downloading from the pinned upstream release tag. Falls back to the remote download when the local file is missing.
Previously container_app_running_replicas summed revision.replicas across active revisions, which includes replicas the portal shows as Provisioning, Failed, Degraded, Stopped, or Unknown — over-counting live workload. Enumerate the container_apps_revision_replicas subresource per active revision and count only replicas whose running_state is 'Running'. Failed replica-list calls log a warning and skip that revision rather than aborting the run.
ryanjpayne
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five related changes — four to
Azure/azure_cspm_benchmark.py(plus itsrequirements.txt) and one tobenchmark.sh:ResourceManagementClientimport path forazure-mgmt-resource26.0.0.TypeErrorcrash when an AKS agent pool reportscount = None(autoscaler-managed or virtual-node pools).AgentPool.countlookup with direct VMSS-instance enumeration for accurate node counts across all pool types.container_app_replicascolumn that counts only actually-running replicas across Azure Container Apps in each subscription.benchmark.sh: mirror the AWS branch behavior for Azure — use../Azure/azure_cspm_benchmark.pywhen present, otherwise fall back to the remote download.Files changed
Azure/azure_cspm_benchmark.pyAzure/requirements.txtbenchmark.sh1. Import fix:
ResourceManagementClientforazure-mgmt-resource26.0.0Azure/azure_cspm_benchmark.py:20Why: Update Azure code as per release 26.0.0 of
azure-mgmt-resource, same rationale as PR #74. The top-levelazure.mgmt.resourcere-export ofResourceManagementClientwas removed; consumers must import it from the.resourcessubmodule.2. Fix: AKS node-pool crash on
Nonenode countPreviously the script crashed with:
whenever an AKS
AgentPoolreturnedcount = None. This happens for pools where the cluster autoscaler owns scaling and for virtual-node pools — the Azure API leavescountunset in those cases.Superseded by change (3) below, which stops relying on
AgentPool.countentirely.3. Accurate AKS node counting via VMSS enumeration
Rewrote
AzureHandle.container_vmss(aks_resource). It previously listed agent pools viacontainer_client.agent_pools.list(...)and returnedAgentPoolobjects. It now:managed_clusters.get(...)to readnode_resource_group(the auto-generatedMC_<rg>_<cluster>_<region>group).virtual_machine_scale_sets.list(...).virtual_machine_scale_set_vms.list(...).(pool_name, vm_count)tuples. Pool name is resolved from theaks-managed-poolNametag (fallback:poolNametag, then VMSS name).Also:
AzureHandle.compute_client(subscription_id)— anlru_cachedComputeManagementClientfactory.vms_inside_vmssto use it instead of instantiating a fresh client per call.main()step (1) to unpack the new tuple shape.Trade-offs
agent_pools.list(onemanaged_clusters.get+ one VMSS list + one VM list per VMSS). Chattier but accurate.aks_nodes. This matches billing reality (virtual nodes are not VMs); if a future PR wants to surface them, they should be folded intoaci_containersor a separate column.Microsoft.Compute/virtualMachineScaleSets/*/readon the AKS node resource group (the built-inReaderrole covers this).4. New column: Azure Container Apps running-replica count
New dependency
Added
azure-mgmt-appcontainerstoAzure/requirements.txt.New import
New column
container_app_replicas("Container App Replicas") added toheaders,totals, and each per-subscriptionrow. Appears in both the terminaltabulateoutput and theazure-benchmark.csvfile.New
AzureHandlemethodscontainer_apps(subscription_id)container_apps.list_by_subscription().container_app_running_replicas(container_app)container_apps_revision_replicassubresource and counts only replicas whoserunning_state == 'Running'.container_apps_client(subscription_id)lru_cachedContainerAppsAPIClientfactory.Wired into
main()as step (5). Wrapped intry/exceptso subscriptions without theMicrosoft.Appresource provider registered log a warning and continue rather than fail the whole run. Per-revision replica-list callsare individually
try/excepted so throttling or transient 5xx errors on one revision only cost that revision, not the whole run.Semantics
running_state == 'Running'are counted.Provisioning,Failed,Degraded,Stopped, orUnknownare excluded.revision.active == False) are skipped entirely.Naïvely summing
revision.replicas(which is what an earlier draft of this PR did) over-counts live workload because that field is a scheduling count, not a runtime-health count.Caveats
revision.replicas-summing approach for this section: 1 list-revisions per app + 1 list-replicas per active revision. Container Apps ARM has aggressive throttling onlarge tenants; the per-revision
try/exceptkeeps the run alive under partial failure but you may see under-counts logged as warnings when throttled.template.scale.minReplicasper active revision instead.Microsoft.App/jobsare a separate resource type and are intentionally not counted here.Microsoft.App/containerApps/read,Microsoft.App/containerApps/revisions/read,Microsoft.App/containerApps/revisions/replicas/read(the built-inReaderrole covers all three).5.
benchmark.sh: use local Azure script when presentbenchmark.sh:151-172— theAzure|GCP)case now checks for../Azure/azure_cspm_benchmark.py(relative to the./cloud-benchmark/venv, i.e., the repo root) before falling back tocurling from the pinned upstreamrelease tag. When the local file exists,
benchmark.shuses it plus../Azure/requirements.txt, falling back to remote requirements only if the localrequirements.txtis missing. Mirrors the AWS branch pattern exactly.Why: Previously,
benchmark.sh azureignored any local edits inAzure/and always downloadedazure_cspm_benchmark.pyfromhttps://raw.githubusercontent.com/CrowdStrike/cloud-resource-estimator/${RELEASE_VERSION}/Azure/...(currently pinned tov1.0.0). That made it impossible to test local Azure changes end-to-end through the same entrypoint most users invoke.
Scope: Only the Azure branch is affected. GCP still always downloads from remote — the
Azure|GCP)case stays joint, but the new local-file check only fires when$CLOUD == "Azure". Extending it to GCP is a two-linefollow-up if maintainers want it.
Not changed (worth calling out)
vmscolumn. Unchanged from prior behavior and out of scope for this PR — happy to open a follow-up if maintainers want to align with the AWS script's behaviorof separating running from non-running.
main()still keys on theaks-managed-createOperationIDtag. Not changed here — no evidence yet of it missing AKS scale sets in the wild.benchmark.shboth use a hard-coded../AWS/.../../Azure/...relative path that only resolves becausebenchmark.shdoespushd ./cloud-benchmarkbefore callingaudit(). A futurerefactor of that
pushdwould silently break both. A code comment explaining the relative-path assumption would be a worthwhile follow-up, kept out of scope here.Testing
Azure/azure_cspm_benchmark.pydirectly against a subscription with an AKS cluster containing an autoscaler-managed pool (workerpool1) that previously triggered theNoneTypecrash — now completes cleanly andreports the correct instance count.
Running, and excludes replicas inProvisioning/Failed/Degraded/Stopped/Unknown.benchmark.sh azureend-to-end verification is straightforward once merged: from the repo root,./benchmark.sh azureshould printUsing local Azure CSPM benchmark script...before proceeding.