Alert: ACMUWLMetricsCollectorForwardRemoteWriteError

[rbo]

https://github.com/openshift/runbooks/blob/master/alerts/advanced-cluster-management-operator/ACMUWLMetricsCollectorForwardRemoteWriteError.md

[rbo]

% oc logs -f deployment/uwl-metrics-collector-deployment -n open-cluster-management-observability
...
level=error caller=logger.go:60 ts=2026-06-17T06:52:46.145759414Z shard=0 component=forwarder/worker msg="unable to forward results" err="HTTP 409: 3 errors: forwarding request to endpoint {observability-thanos-receive-default-1.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:10901 observability-thanos-receive-default-1.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:19391 }: rpc error: code = AlreadyExists desc = 3 errors: add 3129 series: out of order labels; forwarding request to endpoint {observability-thanos-receive-default-0.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:10901 observability-thanos-receive-default-0.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:19391 }: rpc error: code = AlreadyExists desc = add 133 series: out of order labels; forwarding request to endpoint {observability-thanos-receive-default-2.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:10901 observability-thanos-receive-default-2.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:19391 }: rpc error: code = AlreadyExists desc = add 120 series: out of order labels; forwarding request to endpoint {observability-thanos-receive-default-1.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:10901 observability-thanos-receive-default-1.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:19391 }: rpc error: code = AlreadyExists desc = add 3269 series: out of order labels; forwarding request to endpoint {observability-thanos-receive-default-2.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:10901 observability-thanos-receive-default-2.observability-thanos-receive-default.open-cluster-management-observability.svc.cluster.local:19391 }: rpc error: code = AlreadyExists desc = add 1974 series: out of order labels\n"

=> rpc error: code = AlreadyExists desc
=> Out of order of labels or sample

[rbo]

Let's check the time in the cluster:

% oc get nodes -o name | xargs -t -n1 -INODE oc debug -q NODE -- /host/bin/chronyc tracking
oc debug -q node/inf4 -- /host/bin/chronyc tracking
Reference ID    : D484611A (vps-ber1.orleans.ddnss.de)
Stratum         : 3
Ref time (UTC)  : Wed Jun 17 06:42:28 2026
System time     : 0.000213970 seconds slow of NTP time
Last offset     : -0.000034722 seconds
RMS offset      : 0.000070281 seconds
Frequency       : 24.375 ppm slow
Residual freq   : -0.001 ppm
Skew            : 0.015 ppm
Root delay      : 0.020350575 seconds
Root dispersion : 0.001296449 seconds
Update interval : 1024.2 seconds
Leap status     : Normal
oc debug -q node/inf5 -- /host/bin/chronyc tracking
Reference ID    : 55A3A8E3 (z.cincura.net)
Stratum         : 2
Ref time (UTC)  : Wed Jun 17 06:43:53 2026
System time     : 0.000256770 seconds slow of NTP time
Last offset     : -0.000179781 seconds
RMS offset      : 0.000095676 seconds
Frequency       : 20.360 ppm slow
Residual freq   : -0.001 ppm
Skew            : 0.023 ppm
Root delay      : 0.019387513 seconds
Root dispersion : 0.001089712 seconds
Update interval : 1042.7 seconds
Leap status     : Normal
oc debug -q node/inf6 -- /host/bin/chronyc tracking
Reference ID    : A29FC87B (time.cloudflare.com)
Stratum         : 4
Ref time (UTC)  : Wed Jun 17 06:55:39 2026
System time     : 0.000113121 seconds slow of NTP time
Last offset     : -0.000045148 seconds
RMS offset      : 0.000054455 seconds
Frequency       : 13.944 ppm slow
Residual freq   : +0.001 ppm
Skew            : 0.012 ppm
Root delay      : 0.016761152 seconds
Root dispersion : 0.000656537 seconds
Update interval : 1028.5 seconds
Leap status     : Normal
oc debug -q node/storm2 -- /host/bin/chronyc tracking
Reference ID    : A29FC801 (time.cloudflare.com)
Stratum         : 4
Ref time (UTC)  : Wed Jun 17 06:58:31 2026
System time     : 0.000411620 seconds fast of NTP time
Last offset     : +0.000023571 seconds
RMS offset      : 0.000107047 seconds
Frequency       : 1.895 ppm slow
Residual freq   : +0.001 ppm
Skew            : 0.022 ppm
Root delay      : 0.016674671 seconds
Root dispersion : 0.000582892 seconds
Update interval : 1036.1 seconds
Leap status     : Normal
oc debug -q node/storm3 -- /host/bin/chronyc tracking
Reference ID    : A29FC87B (time.cloudflare.com)
Stratum         : 4
Ref time (UTC)  : Wed Jun 17 06:44:11 2026
System time     : 0.000214291 seconds fast of NTP time
Last offset     : -0.000003223 seconds
RMS offset      : 0.000035708 seconds
Frequency       : 9.515 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.007 ppm
Root delay      : 0.016724225 seconds
Root dispersion : 0.001142942 seconds
Update interval : 1037.8 seconds
Leap status     : Normal
oc debug -q node/storm5 -- /host/bin/chronyc tracking
Reference ID    : BC4422AD (srv01-nc.securepod.org)
Stratum         : 3
Ref time (UTC)  : Wed Jun 17 06:51:11 2026
System time     : 0.000105987 seconds fast of NTP time
Last offset     : +0.000098893 seconds
RMS offset      : 0.000135146 seconds
Frequency       : 2.147 ppm fast
Residual freq   : +0.000 ppm
Skew            : 0.041 ppm
Root delay      : 0.020885698 seconds
Root dispersion : 0.003767199 seconds
Update interval : 1029.8 seconds
Leap status     : Normal
oc debug -q node/storm6 -- /host/bin/chronyc tracking
Reference ID    : A29FC801 (time.cloudflare.com)
Stratum         : 4
Ref time (UTC)  : Wed Jun 17 06:59:45 2026
System time     : 0.000091427 seconds fast of NTP time
Last offset     : +0.000016054 seconds
RMS offset      : 0.000035266 seconds
Frequency       : 2.905 ppm fast
Residual freq   : +0.000 ppm
Skew            : 0.007 ppm
Root delay      : 0.016909976 seconds
Root dispersion : 0.000245261 seconds
Update interval : 1024.9 seconds
Leap status     : Normal
oc debug -q node/ucs55 -- /host/bin/chronyc tracking
Reference ID    : A29FC87B (time.cloudflare.com)
Stratum         : 4
Ref time (UTC)  : Wed Jun 17 06:46:24 2026
System time     : 0.000145354 seconds slow of NTP time
Last offset     : -0.000040832 seconds
RMS offset      : 0.000100792 seconds
Frequency       : 57.206 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.011 ppm
Root delay      : 0.017239731 seconds
Root dispersion : 0.001087061 seconds
Update interval : 1041.8 seconds
Leap status     : Normal
oc debug -q node/ucs56 -- /host/bin/chronyc tracking
Reference ID    : A29FC801 (time.cloudflare.com)
Stratum         : 4
Ref time (UTC)  : Wed Jun 17 06:53:37 2026
System time     : 0.000022358 seconds fast of NTP time
Last offset     : +0.000004048 seconds
RMS offset      : 0.000024277 seconds
Frequency       : 18.233 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.008 ppm
Root delay      : 0.017062431 seconds
Root dispersion : 0.000707957 seconds
Update interval : 1043.7 seconds
Leap status     : Normal
oc debug -q node/ucs57 -- /host/bin/chronyc tracking
Reference ID    : D484611A (vps-ber1.orleans.ddnss.de)
Stratum         : 3
Ref time (UTC)  : Wed Jun 17 06:56:13 2026
System time     : 0.000261457 seconds slow of NTP time
Last offset     : -0.000103464 seconds
RMS offset      : 0.000155269 seconds
Frequency       : 36.788 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.018 ppm
Root delay      : 0.020424195 seconds
Root dispersion : 0.000986776 seconds
Update interval : 1034.7 seconds
Leap status     : Normal

[rbo]

https://access.redhat.com/solutions/7143284

[rbo]

https://access.redhat.com/solutions/7142419

Looks like a know ACM Issue, 2.16.0 is installed

[rbo]

Upgrade approved to 2.16.2

[rbo]

With the upgrade, the alert is gone.