fix: handle azure keyvault throttling with retry logic

[thomas-rabiller]

Include retry logic to deal with Azure KeyVault throttling errors that may occur during large sign batches, as described in Microsoft docs:
https://learn.microsoft.com/en-us/azure/key-vault/general/overview-throttling

[thomas-rabiller]

#135

[PhonicUK]

This also looks like a solution for #249

[fabricioferreira]

I am looking forward to this PR getting approved.

[fabricioferreira]

Changes looks good.

[fabricioferreira]

Hi @vcsjones , any reason not to proceed with this merge? Azure's key vault throttling is insane and keeps failing our pipelines. Thanks!

[casuffitsharp]

I believe this issue was addressed in #251. It’s included in v6.0.0-beta1 (check the release announcement here: https://github.com/vcsjones/AzureSignTool/releases/tag/v6.0.0-beta1).

[larde]

We still have this issue on 6.0.1

[ldaniels-dennemeyer]

Me too, retry mechanism does not seem to work.

[fedorbirjukov]

The retry mechanism is indeed part of 6.0.0. See release notes or the notes of the beta mentioned above (link: #271):

Concurrency

By far one issue that gets reported that most are failures from excessive concurrency. There are two parts that can result failures from throttling. The first is Azure. Azure limits the number of operations a KeyVault can perform in a given period. The second is timestamping from a Certificate Authority.

There are two changes in this release to help with this.

1. The --max-degree-of-parallelism (-mdop) now defaults to 4, meaning no more than 4 sign operations will happen concurrently. To revert back to the previous behavior, set --max-degree-of-parallelism to 0, or a higher number if you still want some limit on concurrent operations.
2. The signing operation will now retry failed signing operations if the failure was the result of Azure performing throttling.

So, it must be the CA throttling that limits the signing throughput.

This PR should be abandoned.

[bdagnin]

For us it does seem to be the timestamp service that's causing throttling failures. The Azure retry mechanism in v6.0.1 doesn't kick in at all for these.

I've pulled v6.x into this change and created a build here: https://github.com/bdagnin/AzureSignTool/releases/download/v6.0.2-alpha.1.retry/AzureSignTool.exe
This and -mdop 1 has solved our signing failures.