From a314a1ee1cfcaed5fab56f08e5dfa15a085db893 Mon Sep 17 00:00:00 2001 From: Prachiti Talgulkar Date: Fri, 12 Jun 2026 11:01:11 +0530 Subject: [PATCH 1/3] MCO: fix arm64 arm64-periodics jobs failing due to incompatible image arch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mco-conf-day2-add-mcoqe-robot-to-pull-secret step used `from: upi-installer` which resolves to ocp_5.0_upi-installer — an x86_64-only image. On arm64 periodic jobs the multiarch-tuning-operator detects no supported architectures in common and sets supportedArchitectures:{}, making the pod permanently unschedulable. This has been causing all MCO arm64 periodic jobs to fail at this step before any tests run. Switch to origin/centos:8 which is a multi-arch manifest list (already used by other steps in the same workflow) and provides the jq/bash/base64 tools the step script needs. oc is already injected via cli:latest. Co-Authored-By: Claude Sonnet 4.6 --- .../mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml index 163d87fb23181..016f71e3b8b81 100644 --- a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml +++ b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml @@ -1,6 +1,9 @@ ref: as: mco-conf-day2-add-mcoqe-robot-to-pull-secret - from: upi-installer + from_image: + namespace: origin + name: centos + tag: "8" cli: latest commands: mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh resources: From 1a1581b852c5bfdaa023f84b77230893a2dc7153 Mon Sep 17 00:00:00 2001 From: Prachiti Talgulkar Date: Fri, 12 Jun 2026 11:35:29 +0530 Subject: [PATCH 2/3] MCO: fix arm64 periodic jobs failing due to incompatible upi-installer image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mco-conf-day2-add-mcoqe-robot-to-pull-secret step used `from: upi-installer` which resolves to ocp_5.0_upi-installer — an x86_64-only image. On arm64 periodic jobs the multiarch-tuning-operator detects no supported architectures in common and sets supportedArchitectures:{}, making the pod permanently unschedulable. This has caused all MCO arm64 periodic jobs to fail before any tests run. The same step runs fine on amd64 GCP jobs (e.g. e2e-gcp-mco-disruptive- techpreview-3of3 succeeded in 3m22s with the same image). Switch to ocp/4.18:upi-installer which is a multi-arch image with the same toolset (jq, bash, base64) and is already used successfully in the same arm64 workflow (ipi-conf-telemetry step). Co-Authored-By: Claude Sonnet 4.6 --- .../mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml index 016f71e3b8b81..8d09d98d6d08d 100644 --- a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml +++ b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml @@ -1,9 +1,9 @@ ref: as: mco-conf-day2-add-mcoqe-robot-to-pull-secret from_image: - namespace: origin - name: centos - tag: "8" + namespace: ocp + name: "4.18" + tag: upi-installer cli: latest commands: mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh resources: From 037c4911d7cd9f096d515379e114b82298d530a6 Mon Sep 17 00:00:00 2001 From: Prachiti Talgulkar Date: Fri, 12 Jun 2026 13:59:04 +0530 Subject: [PATCH 3/3] MCO: fix arm64 periodic jobs - replace upi-installer with cli + python3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mco-conf-day2-add-mcoqe-robot-to-pull-secret step used `from: upi-installer` which resolves to ocp_5.0_upi-installer (x86_64-only). On arm64 jobs, multiarch-tuning-operator sets supportedArchitectures:{} making the pod permanently unschedulable — no tests ever ran. Fix: - Switch from `from: upi-installer` to `from: cli` so the image is resolved from each job's own release payload at the correct architecture. This works for all release streams (4.19/4.20/4.21/4.22/5.0/5.1/main) without version pinning. - Replace `jq -s '.[0] * .[1]'` with a python3 deep-merge (python3 is available in the cli image) to remove the jq dependency. Confirmed: the same step runs fine in amd64 GCP jobs (e2e-gcp-mco- disruptive-techpreview-3of3 succeeded in 3m22s), proving the script logic is correct. Only the base image lacks arm64 support. Co-Authored-By: Claude Sonnet 4.6 --- ...f-day2-add-mcoqe-robot-to-pull-secret-commands.sh | 12 +++++++++++- ...conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml | 5 +---- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh index 97178112804aa..3064d72b94daf 100644 --- a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh +++ b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh @@ -28,7 +28,17 @@ echo -n '{"auths": {"quay.io/mcoqe": {"auth": "'"$(base64 -w 0 /var/run/vault/mc # Add the mcoqe credentials to the custer's pull-secret echo "Merge mcoqe credentials and the global cluster pull secret" -jq -s '.[0] * .[1]' "$cluster_pull_secret_file" "$mcoqe_pull_secret_file" > "$merged_pull_secret_file" +python3 -c " +import json, sys +def deep_merge(a, b): + r = dict(a) + for k, v in b.items(): + r[k] = deep_merge(a.get(k, {}), v) if isinstance(v, dict) else v + return r +with open(sys.argv[1]) as f: a = json.load(f) +with open(sys.argv[2]) as f: b = json.load(f) +print(json.dumps(deep_merge(a, b))) +" "$cluster_pull_secret_file" "$mcoqe_pull_secret_file" > "$merged_pull_secret_file" # Update the cluster's pull-secret with the new value echo "Update the global cluster pull secret with the new merged credentials" diff --git a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml index 8d09d98d6d08d..ea954c490a069 100644 --- a/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml +++ b/ci-operator/step-registry/mco/conf/day2/add-mcoqe-robot-to-pull-secret/mco-conf-day2-add-mcoqe-robot-to-pull-secret-ref.yaml @@ -1,9 +1,6 @@ ref: as: mco-conf-day2-add-mcoqe-robot-to-pull-secret - from_image: - namespace: ocp - name: "4.18" - tag: upi-installer + from: cli cli: latest commands: mco-conf-day2-add-mcoqe-robot-to-pull-secret-commands.sh resources: