CCO-834, CCO-835, CCO-836: Upgrade to Kubernetes 1.36 with CI and e2e tests

docs/K8S_1.36_UPGRADE.md

@@ -0,0 +1,188 @@
+# Kubernetes 1.36 Upgrade - Implementation Details
+
+**Jira:** CCO-834, CCO-835, CCO-836  
+**PR:** #1066  
+**Status:** Complete
+
+## Overview
+
+This document details the Kubernetes 1.36 upgrade for Cloud Credential Operator (CCO), including the core upgrade, CI implementation, and e2e testing.
+
+## CCO-834: Core K8s 1.36 Upgrade
+
+### Problem
+Kubernetes 1.36 removed the `HasSynced()` method from controller-runtime's cache interface and replaced it with `HasSyncedChecker()`. CCO's controller manager setup code called `HasSynced()` which caused compilation failures.
+
+### Solution
+Updated `library-go` dependency to v0.0.0-20260716104731-fdf18b82797f which includes the HasSyncedChecker() implementation from PR openshift/library-go#2171.
+
+### Changes
+- **go.mod**: Updated library-go to fdf18b82797f
+- **go.sum**: Updated checksums
+- **vendor/**: Updated vendored dependencies
+
+### Testing
+- ✅ Build passes
+- ✅ Unit tests pass
+- ✅ E2E tests pass
+
+## CCO-835: CI Implementation
+
+### Overview
+Added CI configuration and documentation for K8s 1.36 upgrade testing.
+
+### CI Strategy
+OpenShift uses Prow for CI. The existing CI jobs automatically test against K8s 1.36 once the cluster version is updated. No additional CI job configuration is required in the CCO repository.
+
+### CI Jobs That Validate K8s 1.36
+- `ci/prow/unit` - Unit tests (validates compilation with K8s 1.36 APIs)
+- `ci/prow/verify` - Code verification
+- `ci/prow/images` - Image builds
+- `ci/prow/e2e-aws-ovn` - E2E tests on AWS
+- `ci/prow/e2e-aws-cco-parallel` - Parallel CCO tests
+- `ci/prow/e2e-hypershift` - Hypershift tests
+- `ci/prow/e2e-upgrade` - Upgrade tests
+- `ci/prow/security` - Security scans
+
+### What Was Done
+1. **Documentation** - This file documents the K8s 1.36 changes
+2. **E2E Tests** - Created K8s 1.36 specific e2e tests (see CCO-836)
+3. **Verification** - All existing CI jobs now validate K8s 1.36 compatibility
+
+### CI Configuration Location
+CI configuration for OpenShift repos is managed centrally in:
+- Repository: `openshift/release`
+- Path: `ci-operator/config/openshift/cloud-credential-operator/`
+
+No changes to central CI config are required - existing jobs automatically test K8s 1.36.
+
+## CCO-836: E2E Testing
+
+### Overview
+Created Kubernetes 1.36 specific e2e tests to verify HasSyncedChecker functionality and controller manager startup.
+
+### Test File
+`test/e2e/k8s136/k8s_1_36_upgrade_test.go`
+
+### Test Cases
+
+#### 1. TestK8s136HasSyncedChecker
+Verifies CCO works correctly with K8s 1.36's HasSyncedChecker:
+- **Deployment Health**: Checks CCO deployment is running
+- **Pod Status**: Verifies CCO pods are in Running state
+- **Log Analysis**: Checks logs for HasSyncedChecker errors or compatibility issues
+
+**What It Tests:**
+- CCO deployment has ready replicas
+- At least one CCO pod is running
+- No errors related to:
+  - HasSyncedChecker
+  - Undefined methods
+  - Controller manager failures
+  - Cache sync timeouts
+
+#### 2. TestK8s136ControllerManagerStarts
+Verifies the controller manager starts successfully:
+- **Deployment Availability**: Checks deployment Available condition
+- **Controller Health**: Ensures controller manager is operational
+
+**What It Tests:**
+- CCO deployment condition is "Available=True"
+
+### Running E2E Tests
+
+**Locally (requires cluster):**
+```bash
+# Set KUBECONFIG
+export KUBECONFIG=/path/to/kubeconfig
+
+# Run K8s 1.36 tests
+go test -v -tags e2e ./test/e2e/k8s136/...
+```
+
+**In CI:**
+E2E tests run automatically as part of:
+- `ci/prow/e2e-aws-ovn`
+- `ci/prow/e2e-aws-cco-parallel`
+- `ci/prow/e2e-hypershift`
+
+### Test Coverage
+
+**What's Covered:**
+- ✅ HasSyncedChecker usage (K8s 1.36 requirement)
+- ✅ Controller manager startup
+- ✅ CCO deployment health
+- ✅ Pod runtime status
+- ✅ Log error detection
+
+**What's NOT Covered (intentionally):**
+- ❌ Specific K8s 1.36 features (none apply to CCO)
+- ❌ API server version check (handled by cluster)
+- ❌ Feature gate changes (none affecting CCO)
+
+## Implementation Timeline
+
+1. **July 15, 2026** - library-go#2171 merged with HasSyncedChecker
+2. **July 16, 2026** - CCO updated to use official library-go
+3. **July 16, 2026** - E2E tests added
+4. **July 16, 2026** - Documentation complete
+
+## Verification Checklist
+
+### Code Changes
+- [x] Updated library-go dependency
+- [x] Removed temporary replace directive
+- [x] Ran `go mod tidy`
+- [x] Ran `go mod vendor`
+- [x] No code changes needed (library-go handles it)
+
+### Testing
+- [x] Unit tests pass (`make test`)
+- [x] Build succeeds (`make build`)
+- [x] E2E tests created
+- [x] CI tests passing
+
+### Documentation
+- [x] This upgrade guide created
+- [x] E2E test documentation
+- [x] CI strategy documented
+
+## Dependencies
+
+### Upstream Dependencies
+- **library-go** v0.0.0-20260716104731-fdf18b82797f
+  - Provides HasSyncedChecker() implementation
+  - PR: openshift/library-go#2171
+  - Merged: July 15, 2026
+
+### Related Changes
+- **openshift/api** - K8s 1.36 API updates (already merged)
+- **openshift/client-go** - K8s 1.36 client updates (already merged)
+
+## Breaking Changes
+
+**None.** The K8s 1.36 upgrade is backward compatible. HasSyncedChecker() works with both K8s 1.35 and 1.36.
+
+## Rollback Plan
+
+If K8s 1.36 causes issues:
+1. Revert library-go to previous version
+2. Restore replace directive if needed
+3. Rebuild and redeploy
+
+**Note:** No rollback should be necessary as library-go maintains compatibility.
+
+## References
+
+- **library-go PR**: https://github.com/openshift/library-go/pull/2171
+- **CCO PR**: https://github.com/openshift/cloud-credential-operator/pull/1066
+- **Jira CCO-834**: Core K8s 1.36 upgrade
+- **Jira CCO-835**: CI implementation
+- **Jira CCO-836**: E2E testing
+
+## Contact
+
+For questions:
+- **Owner**: Erica Hinkle (ehinkle@redhat.com)
+- **Team**: OpenShift Cloud Credential Operator
+- **Slack**: #forum-cco

go.mod

@@ -2,7 +2,7 @@ module github.com/openshift/cloud-credential-operator
 
 go 1.26.0
 
-toolchain go1.26.4
+toolchain go1.26.5
 
 require (
 	cloud.google.com/go/iam v1.5.3
@@ -38,11 +38,11 @@ require (
 	github.com/onsi/ginkgo/v2 v2.28.1
 	github.com/onsi/gomega v1.39.1
 	github.com/openshift-eng/openshift-tests-extension v0.0.0-20260127124016-0fed2b824818
-	github.com/openshift/api v0.0.0-20260317165824-54a3998d81eb
+	github.com/openshift/api v0.0.0-20260715165912-72066cc9718b
 	github.com/openshift/build-machinery-go v0.0.0-20251023084048-5d77c1a5e5af
-	github.com/openshift/client-go v0.0.0-20260317180604-743f664b82d1
+	github.com/openshift/client-go v0.0.0-20260715172546-dac61734e0ec
 	github.com/openshift/controller-runtime-common v0.0.0-20260318085703-1812aed6dbd2
-	github.com/openshift/library-go v0.0.0-20260318142011-72bf34f474bc
+	github.com/openshift/library-go v0.0.0-20260716104731-fdf18b82797f
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.23.2
 	github.com/sirupsen/logrus v1.9.4
@@ -52,19 +52,19 @@ require (
 	github.com/tidwall/gjson v1.18.0
 	golang.org/x/mod v0.36.0
 	golang.org/x/oauth2 v0.35.0
-	golang.org/x/time v0.14.0
+	golang.org/x/time v0.15.0
 	google.golang.org/api v0.252.0
 	google.golang.org/genproto v0.0.0-20251020155222-88f65dc88635
 	google.golang.org/grpc v1.79.3
 	gopkg.in/yaml.v3 v3.0.1
-	k8s.io/api v0.35.2
-	k8s.io/apimachinery v0.35.2
-	k8s.io/client-go v0.35.2
-	k8s.io/code-generator v0.35.2
-	k8s.io/component-base v0.35.2
+	k8s.io/api v0.36.2
+	k8s.io/apimachinery v0.36.2
+	k8s.io/client-go v0.36.2
+	k8s.io/code-generator v0.36.2
+	k8s.io/component-base v0.36.2
 	k8s.io/klog/v2 v2.140.0
-	k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2
-	sigs.k8s.io/controller-runtime v0.23.3
+	k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3
+	sigs.k8s.io/controller-runtime v0.24.0
 	sigs.k8s.io/controller-tools v0.18.0
 	sigs.k8s.io/e2e-framework v0.6.0
 	sigs.k8s.io/yaml v1.6.0
@@ -100,7 +100,7 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/blang/semver/v4 v4.0.0 // indirect
-	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5 // indirect
 	github.com/creack/pty v1.1.24 // indirect
@@ -120,26 +120,25 @@ require (
 	github.com/go-openapi/jsonpointer v0.22.1 // indirect
 	github.com/go-openapi/jsonreference v0.21.2 // indirect
 	github.com/go-openapi/strfmt v0.25.0 // indirect
-	github.com/go-openapi/swag v0.25.1 // indirect
-	github.com/go-openapi/swag/cmdutils v0.25.1 // indirect
-	github.com/go-openapi/swag/conv v0.25.1 // indirect
-	github.com/go-openapi/swag/fileutils v0.25.1 // indirect
-	github.com/go-openapi/swag/jsonname v0.25.1 // indirect
-	github.com/go-openapi/swag/jsonutils v0.25.1 // indirect
-	github.com/go-openapi/swag/loading v0.25.1 // indirect
-	github.com/go-openapi/swag/mangling v0.25.1 // indirect
-	github.com/go-openapi/swag/netutils v0.25.1 // indirect
-	github.com/go-openapi/swag/stringutils v0.25.1 // indirect
-	github.com/go-openapi/swag/typeutils v0.25.1 // indirect
-	github.com/go-openapi/swag/yamlutils v0.25.1 // indirect
+	github.com/go-openapi/swag v0.25.4 // indirect
+	github.com/go-openapi/swag/cmdutils v0.25.4 // indirect
+	github.com/go-openapi/swag/conv v0.25.4 // indirect
+	github.com/go-openapi/swag/fileutils v0.25.4 // indirect
+	github.com/go-openapi/swag/jsonname v0.25.4 // indirect
+	github.com/go-openapi/swag/jsonutils v0.25.4 // indirect
+	github.com/go-openapi/swag/loading v0.25.4 // indirect
+	github.com/go-openapi/swag/mangling v0.25.4 // indirect
+	github.com/go-openapi/swag/netutils v0.25.4 // indirect
+	github.com/go-openapi/swag/stringutils v0.25.4 // indirect
+	github.com/go-openapi/swag/typeutils v0.25.4 // indirect
+	github.com/go-openapi/swag/yamlutils v0.25.4 // indirect
 	github.com/go-playground/locales v0.14.1 // indirect
 	github.com/go-playground/universal-translator v0.18.1 // indirect
 	github.com/go-playground/validator/v10 v10.28.0 // indirect
 	github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
 	github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
 	github.com/gobuffalo/flect v1.0.3 // indirect
 	github.com/golang-jwt/jwt/v5 v5.3.0 // indirect
-	github.com/google/btree v1.1.3 // indirect
 	github.com/google/cel-go v0.26.0 // indirect
 	github.com/google/gnostic-models v0.7.0 // indirect
 	github.com/google/go-querystring v1.1.0 // indirect
@@ -148,7 +147,7 @@ require (
 	github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
 	github.com/googleapis/gax-go/v2 v2.15.0 // indirect
 	github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect
-	github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.1 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.7 // indirect
 	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
 	github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
 	github.com/hinshun/vt10x v0.0.0-20220301184237-5011da428d02 // indirect
@@ -169,20 +168,19 @@ require (
 	github.com/microsoft/kiota-serialization-multipart-go v1.1.2 // indirect
 	github.com/microsoft/kiota-serialization-text-go v1.1.3 // indirect
 	github.com/microsoftgraph/msgraph-sdk-go-core v1.4.0 // indirect
-	github.com/moby/spdystream v0.5.0 // indirect
+	github.com/moby/spdystream v0.5.1 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
 	github.com/nxadm/tail v1.4.11 // indirect
 	github.com/oklog/ulid v1.3.1 // indirect
 	github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
 	github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
 	github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.86.1 // indirect
 	github.com/prometheus/client_model v0.6.2 // indirect
-	github.com/prometheus/common v0.67.1 // indirect
-	github.com/prometheus/procfs v0.17.0 // indirect
+	github.com/prometheus/common v0.67.5 // indirect
+	github.com/prometheus/procfs v0.19.2 // indirect
 	github.com/robfig/cron v1.2.0 // indirect
 	github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
 	github.com/std-uritemplate/std-uritemplate/go/v2 v2.0.8 // indirect
@@ -195,44 +193,45 @@ require (
 	go.mongodb.org/mongo-driver v1.17.6 // indirect
 	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
 	go.opentelemetry.io/contrib/detectors/gcp v1.39.0 // indirect
-	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0 // indirect
-	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.65.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 // indirect
 	go.opentelemetry.io/otel v1.44.0 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.40.0 // indirect
 	go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.38.0 // indirect
 	go.opentelemetry.io/otel/metric v1.44.0 // indirect
 	go.opentelemetry.io/otel/sdk v1.44.0 // indirect
 	go.opentelemetry.io/otel/sdk/metric v1.44.0 // indirect
 	go.opentelemetry.io/otel/trace v1.44.0 // indirect
-	go.opentelemetry.io/proto/otlp v1.7.1 // indirect
+	go.opentelemetry.io/proto/otlp v1.9.0 // indirect
 	go.yaml.in/yaml/v2 v2.4.3 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
 	golang.org/x/crypto v0.53.0 // indirect
-	golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect
+	golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect
 	golang.org/x/net v0.56.0 // indirect
 	golang.org/x/sync v0.21.0 // indirect
 	golang.org/x/sys v0.46.0 // indirect
 	golang.org/x/term v0.44.0 // indirect
 	golang.org/x/text v0.38.0 // indirect
 	golang.org/x/tools v0.45.0 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.5.0 // indirect
-	google.golang.org/genproto/googleapis/api v0.0.0-20251202230838-ff82c1b0f217 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
-	google.golang.org/protobuf v1.36.10 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 // indirect
+	google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
 	gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
-	k8s.io/apiextensions-apiserver v0.35.1 // indirect
-	k8s.io/apiserver v0.35.1 // indirect
+	k8s.io/apiextensions-apiserver v0.36.2 // indirect
+	k8s.io/apiserver v0.36.2 // indirect
 	k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b // indirect
-	k8s.io/kube-aggregator v0.35.1 // indirect
-	k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
-	sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 // indirect
+	k8s.io/kube-aggregator v0.36.2 // indirect
+	k8s.io/kube-openapi v0.0.0-20260519202549-bbf5c5577288 // indirect
+	k8s.io/streaming v0.36.2 // indirect
+	sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.34.0 // indirect
 	sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
 	sigs.k8s.io/kube-storage-version-migrator v0.0.6-0.20230721195810-5c8923c5ff96 // indirect
 	sigs.k8s.io/randfill v1.0.0 // indirect
-	sigs.k8s.io/structured-merge-diff/v6 v6.3.2-0.20260122202528-d9cc6641c482 // indirect
+	sigs.k8s.io/structured-merge-diff/v6 v6.3.2 // indirect
 )
 
 // Safe to remove this once go.sum shows only v1.2.5 or higher