Add new tooling to cri.py to support modifying kubelet using a daemonset with predefined custom kubelet flags by pdamianov-dev · Pull Request #1034 · Azure/telescope

pdamianov-dev · 2026-01-23T20:25:59Z

This is related to the creation of a suitable framework to run pressure tests against node images as part of Node Hardening efforts.

Supporting documents:

…et in engine steps

Copilot

Pull request overview

Adds a new “CRI pressure” execution path that can modify kubelet flags via a privileged DaemonSet before running ClusterLoader2, enabling node pressure testing for node hardening work.

Changes:

Switches the k8s-resource-pressure topology to use a new cri-pressure engine template.
Adds a new pipeline step template that applies a kubelet-modifying DaemonSet before running the benchmark.
Extends cri.py with a modify-kubelet subcommand and adds DaemonSet-manifest generation in clusterloader2/utils.py.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

File	Description
steps/topology/k8s-resource-pressure/execute-clusterloader2.yml	Routes the topology to the new `cri-pressure` execution template.
steps/engine/clusterloader2/cri-pressure/execute.yml	Adds a pre-benchmark step to apply a kubelet config updater DaemonSet, then runs CL2.
modules/python/clusterloader2/utils.py	Adds DaemonSet YAML generation for updating kubelet flags (but also modifies imports/initialization).
modules/python/clusterloader2/cri/cri.py	Adds `modify-kubelet` CLI command and wires it to the DaemonSet generator.

steps/engine/clusterloader2/cri-pressure/execute.yml

Copilot · 2026-01-23T20:32:41Z

modules/python/clusterloader2/cri/cri.py

+    parser_modify_kubelet.add_argument(
+        "--custom_kubelet_config", type=str, help="Custom kubelet flags in string format"
+    )


--custom_kubelet_config is optional, but modify-kubelet depends on it to generate the DaemonSet. Make this argument required (or validate non-empty and exit with a clear error) to avoid applying a manifest with an empty/None flags string.

Copilot · 2026-01-23T20:32:41Z

modules/python/clusterloader2/cri/cri.py

+    # Sub-command for modify-kubelet
+    parser_modify_kubelet = subparsers.add_parser(
+        "modify-kubelet", help="Add custom flags to kubelet and apply via daemonset"
+    )
+    parser_modify_kubelet.add_argument(


There are existing unit tests covering main() subcommands in modules/python/tests/test_cri.py, but the new modify-kubelet subcommand isn’t covered. Add a test that exercises arg parsing and verifies it invokes the right handler (and ideally validates the manifest output path).

Copilot · 2026-01-23T20:32:41Z

modules/python/clusterloader2/utils.py

 from xml.dom import minidom
 import json
 import os
-import docker
-from clients.docker_client import DockerClient
-from utils.logger_config import get_logger, setup_logging

-setup_logging()
-logger = get_logger(__name__)
+


The imports / module-level initialization for Docker + logging were removed, but this module still references DockerClient, docker.errors.ContainerError, and logger (e.g., in run_cl2_command and process_cl2_reports). This will raise NameError at runtime when running clusterloader2. Re-introduce the needed imports and define logger (or switch these functions to use logging.getLogger(__name__) and ensure docker/DockerClient are imported).

Copilot · 2026-01-23T20:32:42Z

modules/python/clusterloader2/utils.py

+      labels:
+        app: kubelet-config-updater
+    spec:
+      hostPID: true
+      containers:


This DaemonSet is privileged with host mounts but has no node scoping. As written it can run on every node and modify kubelet configuration cluster-wide. Add a nodeSelector/affinity (e.g., cri-resource-consume=true) and at minimum constrain to Linux nodes to reduce blast radius.

modules/python/clusterloader2/utils.py

…/telescope into pdamianov-dev/k8s-pressure

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

pipelines/perf-eval/CRI Benchmark/k8s-resource-pressure.yml

vittoriasalim · 2026-02-03T06:14:40Z

modules/python/clusterloader2/cri/cri.py

        "--registry_info", type=str, help="Container registry information scraped",
    )

+    # Sub-command for modify-kubelet


if you need to apply daemonset before benchmarking start, you can apply it at validation stage rather than in the engine e.g : https://github.com/Azure/telescope/pull/1045/changes
https://github.com/Azure/telescope/blob/main/steps/topology/karpenter/validate-resources.yml

I am not exactly sure what is this for
if you need to modify the node image, can do something like this https://github.com/Azure/telescope/pull/1044/changes

vittoriasalim · 2026-02-03T06:15:08Z

modules/python/clusterloader2/utils.py

+kind: DaemonSet
+metadata:
+  name: kubelet-config-updater
+  namespace: kube-system


can we move this to validation stage , e.g: https://github.com/Azure/telescope/pull/1045/changes

Yea I can take a look at this, my only real concern is how to pass the parameters in as the yaml itself, and will let you know if I can have the same effect

We have this example where it patch the VM_SIZE to yaml template

telescope/steps/topology/karpenter/validate-resources.yml

Line 28 in 39223da

if [ -n "${VM_SIZE:-}" ]; then

…/telescope into pdamianov-dev/k8s-pressure

Removed the modify_kubelet_clusterloader2 function and its associated functionality.

vittoriasalim · 2026-02-06T02:48:18Z

pipelines/perf-eval/CRI Benchmark/k8s-resource-pressure.yml

@@ -1,8 +1,34 @@
 trigger: none

+parameters:


We do not need parameters , simply enable them in your test

n1-p60-memory-managed: node_count: 1 max_pods: 60 repeats: 1 operation_timeout: 3m load_type: memory pod_startup_latency_threshold: 23s kubernetes_version: "1.34" k8s_os_disk_type: Managed scrape_kubelets: True enable_custom_kubelet: false kubelet_config_type: "eviction-hard" # eviction-soft, eviction-soft-grace-period

in validate-resources

can call
$KUBELET_CONFIG_TYPE

if KUBELET_CONFIG_TYPE == ëviction-hard :

override the memory, nodefs and pid
else if
else if
so on and so on

What if we want to pass in different flags with each run without changing the pipeline definition?

Our intention may not be to schedule this pipeline, we will most likely do it manually or have a separate pipeline trigger

what if i set a queue time variable instead?

vittoriasalim

remove parameters and only enable those variable inside your test

rename use_custom_kubelet to enable_custom_kubelet

pdamianov-dev added 4 commits January 21, 2026 20:01

new daemonset for kubelet flag change

ed237de

testing pipeline param

e1d140c

adding daemonset support with passed in string param

9dc534c

full e2e workflow of getting kubelet flags and applying them to kubel…

b388403

…et in engine steps

pdamianov-dev requested review from alyssa1303 and anson627 as code owners January 23, 2026 20:26

Copilot AI review requested due to automatic review settings January 23, 2026 20:26

pdamianov-dev requested review from sumanthreddy29, vittoriasalim, wonderyl and xinWeiWei24 as code owners January 23, 2026 20:26

Copilot started reviewing on behalf of pdamianov-dev January 23, 2026 20:26 View session

Copilot AI reviewed Jan 23, 2026

View reviewed changes

pdamianov-dev and others added 14 commits January 28, 2026 16:15

testing convertToJson

1e7f30c

test fixes

5f08dde

modify to pass in json from object in params

f9b41ab

Merge branch 'pdamianov-dev/k8s-pressure' of https://github.com/Azure…

39e417b

…/telescope into pdamianov-dev/k8s-pressure

fix string syntax error in utils.py

ae7f32a

fix unbound python path error

2c78447

moved pythonpath to separate line

03c6fb5

remove -u as its not anywhere else

2113fc6

re-add imports

013ab43

re add kubectl apply

a6812f6

pylint errors

7b5526b

yaml lint errors

1f97096

Apply suggestions from code review

7ee9cfc

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update execute.yml

534ad42

vittoriasalim reviewed Feb 3, 2026

View reviewed changes

pipelines/perf-eval/CRI Benchmark/k8s-resource-pressure.yml Show resolved Hide resolved

vittoriasalim reviewed Feb 3, 2026

View reviewed changes

pdamianov-dev and others added 15 commits February 4, 2026 20:12

use helm chart and jq for json processing in validation yml

bf219bf

stop using cri-pressure engine

5917491

Merge branch 'pdamianov-dev/k8s-pressure' of https://github.com/Azure…

747b72e

…/telescope into pdamianov-dev/k8s-pressure

ensure full string for flags

98a6b8a

temporary comment out for testing

e5ad660

more fixes

8557191

Merge branch 'pdamianov-dev/k8s-pressure' of https://github.com/Azure…

37cd36c

…/telescope into pdamianov-dev/k8s-pressure

check pod status after helm upgrade

e515461

steps/topology/k8s-resource-pressure/validate-resources.yml

df0da52

try with file

0d33331

switch to rel path

5ba9069

fixed kubelet flags issue

4237ad2

revert all python changes

1b4d651

Remove modify_kubelet_clusterloader2 function

e4b9bd1

Removed the modify_kubelet_clusterloader2 function and its associated functionality.

Remove trailing newline from utils.py

e3166e0

vittoriasalim reviewed Feb 6, 2026

View reviewed changes

vittoriasalim requested changes Feb 6, 2026

View reviewed changes

Conversation

pdamianov-dev commented Jan 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vittoriasalim Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vittoriasalim Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pdamianov-dev Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

vittoriasalim Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

vittoriasalim Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

pdamianov-dev Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

pdamianov-dev Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

pdamianov-dev Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

vittoriasalim left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vittoriasalim Feb 3, 2026 •

edited

Loading

vittoriasalim Feb 3, 2026 •

edited

Loading

vittoriasalim left a comment •

edited

Loading