-
Notifications
You must be signed in to change notification settings - Fork 150
Improve CI resilency against connectivity errors #7974
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (7974) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (69ms) : 67, 70
master - mean (68ms) : 66, 70
section Bailout
This PR (7974) - mean (72ms) : 71, 73
master - mean (72ms) : 71, 73
section CallTarget+Inlining+NGEN
This PR (7974) - mean (1,016ms) : 958, 1073
master - mean (1,006ms) : 968, 1044
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (106ms) : 104, 109
master - mean (106ms) : 103, 108
section Bailout
This PR (7974) - mean (107ms) : 106, 108
master - mean (107ms) : 105, 108
section CallTarget+Inlining+NGEN
This PR (7974) - mean (717ms) : 679, 754
master - mean (712ms) : 678, 746
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (94ms) : 92, 96
master - mean (94ms) : 92, 96
section Bailout
This PR (7974) - mean (95ms) : 94, 96
master - mean (94ms) : 93, 95
section CallTarget+Inlining+NGEN
This PR (7974) - mean (669ms) : 649, 689
master - mean (667ms) : 643, 690
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (93ms) : 90, 95
master - mean (92ms) : 90, 94
section Bailout
This PR (7974) - mean (93ms) : 92, 95
master - mean (93ms) : 92, 94
section CallTarget+Inlining+NGEN
This PR (7974) - mean (632ms) : 618, 646
master - mean (630ms) : 614, 646
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (194ms) : 186, 202
master - mean (194ms) : 190, 199
section Bailout
This PR (7974) - mean (197ms) : 195, 200
master - mean (197ms) : 194, 200
section CallTarget+Inlining+NGEN
This PR (7974) - mean (1,121ms) : 1061, 1181
master - mean (1,117ms) : 1064, 1170
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (277ms) : 273, 282
master - mean (278ms) : 272, 284
section Bailout
This PR (7974) - mean (279ms) : 273, 284
master - mean (278ms) : 274, 282
section CallTarget+Inlining+NGEN
This PR (7974) - mean (904ms) : 859, 949
master - mean (909ms) : 867, 950
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (270ms) : 264, 277
master - mean (271ms) : 266, 277
section Bailout
This PR (7974) - mean (270ms) : 267, 273
master - mean (270ms) : 267, 274
section CallTarget+Inlining+NGEN
This PR (7974) - mean (891ms) : 846, 936
master - mean (888ms) : 847, 929
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (7974) - mean (269ms) : 264, 274
master - mean (271ms) : 265, 277
section Bailout
This PR (7974) - mean (269ms) : 265, 273
master - mean (270ms) : 267, 274
section CallTarget+Inlining+NGEN
This PR (7974) - mean (828ms) : 809, 847
master - mean (825ms) : 808, 842
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
BenchmarksBenchmark execution time: 2025-12-19 16:40:49 Comparing candidate commit 8addd7c in PR branch Found 5 performance improvements and 10 performance regressions! Performance is the same for 153 metrics, 18 unstable metrics. scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1
scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0
scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1
scenario:Benchmarks.Trace.HttpClientBenchmark.SendAsync net6.0
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0
|
Summary of changes
Add retry logic to all external network operations in Azure Pipelines to improve CI reliability and reduce flaky build failures caused by transient network errors. This kind of errors are intermitent. While some jobs failend, very similar ones passed.
GH lately seems to have connection issues like this or this:
This PR improves our pipeline resilency against this type of errors.
Reason for change
Recent CI failures show transient network errors causing build failures:
Unable to connect to the remote serverwhen downloading docker-composeFailed to connect to github.com port 443when cloning repositoriesThese errors are typically transient and succeed on retry. Adding automatic retry logic reduces manual re-runs and improves CI reliability.
Implementation details
Git Operations
git -c http.retry=5 -c http.retryDelay=2to allgit clonecommandsclone-repo.ymlfor both Linux and Windows checkout operationsultimate-pipeline.yml(system-tests clones, master branch fetch)steps/clone-repo.yml(repository checkout operations)Download Operations
Docker Compose (
steps/install-docker-compose-v1.yml)--retry 5 --retry-delay 2 --retry-connrefused --connect-timeout 30 --max-time 120)powershelltopwshand addedInvoke-WebRequestretry parameters (-MaximumRetryCount 5 -RetryIntervalSec 2 -TimeoutSec 120)MSI Downloads (
steps/install-msi.yml)powershelltopwshwith retry parameters.NET Install Script (
steps/install-dotnet-sdk-manually.yml)powershelltopwshwith retry parametersAPI Calls
GitHub API (
ultimate-pipeline.yml)GitHub Status Updates (
steps/update-github-status.yml)Package Installation
ultimate-pipeline.yml, line ~6291)--retries 5 --timeout 120topip install ddapm-test-agentRetry Configuration
All retry configurations use consistent parameters:
Test coverage
pwsh(PowerShell 7+) which is available on all Azure Pipelines agentsOther details