Conversation
delgod
commented
Jul 21, 2026
delgod
commented
Member
- Have you updated relevant documentation?
test_build_and_deploy waited the same hardcoded 600s for both tls variants, but tls_on additionally deploys self-signed-certificates, integrates client-certificates, and performs a rolling sentinel restart to apply the client certificate — all inside that budget. On a loaded CI runner this convergence slips past 600s, producing a flaky "TimeoutError: wait timed out after 600s" at deploy (not in the failover logic). Give the tls_on path 900s; keep tls_off fail-fast at 600s.
Integration artifacts carried only charm-side juju logs; the flaky HA failures (slow sentinel restart on TLS enable, slow promotion after a network cut) live in the valkey-server and sentinel logs on the workload itself. Add a best-effort step that pulls those via Pebble (k8s) or snap (VM) into ~/logs/ so they ship with the existing artifact upload.
…runs K8s integration jobs intermittently time out in test_build_and_deploy with pods stuck `allocating`: a 3-unit deploy requests 9 filesystem PVCs (data/logs/archive x3) and the single-node rawfile-csi provisioner sometimes leaves some Pending for the whole run, so the pods never start. The existing charm-side juju/server-log capture cannot see this -- no pod means no hook ran. Add a K8s-only step to the failure-capture block that dumps PVC, PV, pod, event, storageclass, rawfile-csi controller and node state to ~/logs/k8s-storage-diagnostics.txt. This records which PVCs are Pending and why (provisioner events, node DiskPressure/allocatable, and the storageclass volumeBindingMode) so the next occurrence is diagnosable. It covers every k8s job, runs once at failure, and is best-effort (never fails the job). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…le-csi loop-device race) Canonical K8s's built-in rawfile-csi backs each PV with a loopback device. A 3-unit valkey deploy requests 9 filesystem PVCs at once, and rawfile-csi intermittently races on loop-device allocation (`losetup -f` / `mknod /dev/loopN` fail), leaving a pod's volume unmountable so its charm container crash-loops and the deploy times out at 900s. This is the storage-side of the flaky k8s integration runs; it lands on a random job each run. Install Rancher local-path-provisioner (hostPath-backed, no loop devices -- what the local microk8s test path already uses reliably) in spread prepare-each and route Juju workload storage to it: make `local-path` the default StorageClass and set `workload-storage=local-path` on the testing model (both best-effort). rawfile-csi stays enabled for the single-volume Juju controller, which has no concurrency and so does not hit the race. local-path is WaitForFirstConsumer + Delete, identical semantics to rawfile, so test_storage_reuse (which relies on StatefulSet PVC retention on K8s) is unaffected. Whether Juju honours the default-SC swap or the workload-storage override is only observable on CI; the storage-diagnostics capture added in c099c83 will show which StorageClass the valkey PVCs land on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Capture Valkey server logs" step read `snap logs charmed-valkey.sentinel` (VM) / `pebble logs valkey-sentinel` (k8s), which return the systemd journal / service stdout -- only `systemd Started/Stopped ...service` lifecycle lines. Valkey and Sentinel log to files in the logs volume (config.py sets `logfile` to <log_dir>/valkey.log and <log_dir>/sentinel.log), so the sentinel's +sdown/+odown/+failover/+switch-master events -- the sub-second state needed to diagnose the flaky network-cut/failover timing -- were never captured. A failing vm/test_network_cut_tls_on run confirmed the captured logs held only lifecycle lines and stopped before the network cut. Read the log files directly (k8s: `kubectl exec ... cat`, VM: `juju ssh ... sudo cat`) and keep a separate service-lifecycle capture (`pebble changes` / `snap logs`) for restart-churn context. log_dir paths mirror tests/integration/ha/test_storage_reuse.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.