fix validator onbaording retry logic to avoid failing on archived contracts

[mblaze-da]

Fixes: https://github.com/DACH-NY/cn-test-failures/issues/8335

[mblaze-da]

@martinflorian-da I think this wasn't it.

splitwellValidator app initialization: Initialization failed, so exiting; check the application logs for details
java.lang.RuntimeException: splitwellValidator app initialization: Initialize app failed
	at org.lfdecentralizedtrust.splice.environment.NodeBase.$anonfun$appInitStep$3(NodeBase.scala:225)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:475)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: java.lang.RuntimeException: splitwellValidator app initialization: Initialize node failed
	... 8 more
Caused by: java.lang.RuntimeException: splitwellValidator app initialization: Ensure validator is onboarded failed
	... 8 more
Caused by: org.lfdecentralizedtrust.splice.admin.api.client.commands.HttpCommandException: HTTP 400 Bad Request POST at '/api/sv/v0/onboard/validator' on 127.0.0.1:5114. Command failed, message: UNKNOWN_CONTRACT_SYNCHRONIZERS(9,95125f34): The following contracts have been archived: List(00fcef488c4adb3b7260a6c92b2e445226423bcd9c6dac78a3db3a689702208794ca1212208235f6a004d15e620f80447881aace50546ba6482e18f9adfd97c72c385ddc97) on synchronizer global-domain::122018d37999....
	at org.lfdecentralizedtrust.splice.http.HttpClient$.$anonfun$getApiErrorFromResponse$6(HttpClient.scala:89)
	at scala.util.Success.map(Try.scala:270)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:249)
	at org.apache.pekko.http.scaladsl.util.FastFuture$FulfilledFuture.transform(FastFuture.scala:93)
	at scala.concurrent.Future.map(Future.scala:249)
	at scala.concurrent.Future.map$(Future.scala:249)
	at org.apache.pekko.http.scaladsl.util.FastFuture$FulfilledFuture.map(FastFuture.scala:85)
	at org.lfdecentralizedtrust.splice.http.HttpClient$.getApiErrorFromResponse(HttpClient.scala:89)
	at org.lfdecentralizedtrust.splice.http.HttpClient$.$anonfun$httpClientWithErrors$2(HttpClient.scala:101)
	at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:289)
	at org.lfdecentralizedtrust.splice.http.HttpClient$.$anonfun$httpClientWithErrors$1(HttpClient.scala:105)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:470)
	... 6 more

[martinflorian-da]

@martinflorian-da I think this wasn't it.

splitwellValidator app initialization: Initialization failed, so exiting; check the application logs for details
java.lang.RuntimeException: splitwellValidator app initialization: Initialize app failed
	at org.lfdecentralizedtrust.splice.environment.NodeBase.$anonfun$appInitStep$3(NodeBase.scala:225)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:475)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: java.lang.RuntimeException: splitwellValidator app initialization: Initialize node failed
	... 8 more
Caused by: java.lang.RuntimeException: splitwellValidator app initialization: Ensure validator is onboarded failed
	... 8 more
Caused by: org.lfdecentralizedtrust.splice.admin.api.client.commands.HttpCommandException: HTTP 400 Bad Request POST at '/api/sv/v0/onboard/validator' on 127.0.0.1:5114. Command failed, message: UNKNOWN_CONTRACT_SYNCHRONIZERS(9,95125f34): The following contracts have been archived: List(00fcef488c4adb3b7260a6c92b2e445226423bcd9c6dac78a3db3a689702208794ca1212208235f6a004d15e620f80447881aace50546ba6482e18f9adfd97c72c385ddc97) on synchronizer global-domain::122018d37999....
	at org.lfdecentralizedtrust.splice.http.HttpClient$.$anonfun$getApiErrorFromResponse$6(HttpClient.scala:89)
	at scala.util.Success.map(Try.scala:270)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:249)
	at org.apache.pekko.http.scaladsl.util.FastFuture$FulfilledFuture.transform(FastFuture.scala:93)
	at scala.concurrent.Future.map(Future.scala:249)
	at scala.concurrent.Future.map$(Future.scala:249)
	at org.apache.pekko.http.scaladsl.util.FastFuture$FulfilledFuture.map(FastFuture.scala:85)
	at org.lfdecentralizedtrust.splice.http.HttpClient$.getApiErrorFromResponse(HttpClient.scala:89)
	at org.lfdecentralizedtrust.splice.http.HttpClient$.$anonfun$httpClientWithErrors$2(HttpClient.scala:101)
	at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:289)
	at org.lfdecentralizedtrust.splice.http.HttpClient$.$anonfun$httpClientWithErrors$1(HttpClient.scala:105)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:470)
	... 6 more

@mblaze-da looking at the logs for why this wasn't enough...

We might need to replicate something like this here then also for validator onboarding:

splice/apps/sv/src/main/scala/org/lfdecentralizedtrust/splice/sv/onboarding/joining/JoiningNodeDsoPartyHosting.scala

Line 108 in 1a93eb2

case proposalNotFound: OnboardSvPartyMigrationAuthorizeProposalNotFound =>

[martinflorian-da]

OK I think that this fix is not solving our issue because the retries happen so fast and it all times out so quickly that the SV app doesn't have time to notice the change (i.e., ingestion doesn't happen in time)

2026-05-08T14:59:28.396Z [⋮] INFO - o.l.s.a.a.c.ApiClientRequestLogger:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401---16eb29382f72dd83) - Request (tid:2eb81d72e529a5ecbcc443ab531ed79b) com.daml.ledger.api.v2.CommandService/SubmitAndWait to 127.0.0.1:5101: failed with FAILED_PRECONDITION/UNKNOWN_CONTRACT_SYNCHRONIZERS(9,2eb81d72): The following contracts have been archived: List(00fcef488c4adb3b7260a6c92b2e445226423bcd9c6dac78a3db3a689702208794ca1212208235f6a004d15e620f80447881aace50546ba6482e18f9adfd97c72c385ddc97) on synchronizer global-domain::122018d37999....
2026-05-08T14:59:28.497Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:28.707Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:28.725Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:28.986Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:29.148Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:29.520Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:29.680Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:30.079Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:30.795Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:30.958Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:30.984Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:31.976Z [⋮] INFO - o.l.s.s.a.h.HttpSvPublicHandler:SvFrontendIntegrationTest/config=1d31f8a3/SV=sv1 (31de52b3dda1f12dac1dce527dfac401-onboard_validator-16eb29382f72dd83-86e7b1cefa0da957) - Now retrying operation 'onboard validator via DsoRules'.  
2026-05-08T14:59:32.005Z [⋮] ERROR - o.l.s.v.ValidatorApp:SvFrontendIntegrationTest/config=1d31f8a3/validator=splitwellValidator (31de52b3dda1f12dac1dce527dfac401-app_init--16eb29382f72dd83) - splitwellValidator app initialization: Initialization failed java.lang.RuntimeException: splitwellValidator app initialization: Initialize app failed

Fixing the client side seems the more clean fix after all. I'd also open a new PR for that.

About this PR: on the fence. I think it is more correct, but also with the insight that retries happen so fast I don't think it makes a dent in practice. I'd lean towards not merging.are so short and quick.