docs(contribution): add Crossplane Tuning Knobs guide by gergely-szabo-sap · Pull Request #72 · SAP/crossplane-provider-docs

gergely-szabo-sap · 2026-05-12T13:37:52Z

Add a new contribution guide covering the tunable parameters that control Crossplane provider behavior at scale.

Covers the 5-layer configuration stack (client-go through CLI flags), the --max-reconcile-rate fan-out to MaxConcurrentReconciles, GlobalRateLimiter and QPS/Burst, reconcile flow, backoff, poll interval, leader election, and jitter.

Includes tuning recommendations by landscape size and a provider defaults comparison table (BTP, CF, GCP, Helm, K8s, SQL).

Diagrams: Configuration Stack and --max-reconcile-rate fan-out.

Add a new contribution guide covering the tunable parameters that control Crossplane provider behavior at scale. Covers the 5-layer configuration stack (client-go through CLI flags), the --max-reconcile-rate fan-out to MaxConcurrentReconciles, GlobalRateLimiter and QPS/Burst, reconcile flow, backoff, poll interval, leader election, and jitter. Includes tuning recommendations by landscape size and a provider defaults comparison table (BTP, CF, GCP, Helm, K8s, SQL). Diagrams: Configuration Stack and --max-reconcile-rate fan-out. Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

enrico-kaack-comp

great explanations, really like it. I added some more details from my notes that i took, maybe not all of them are correct we need to verify again.

Regarding the max-reconcile-rate that we incorrectly use for multiple values. I have a PR in BTP open for exposing more of these flags and not use the max-reconcile-rate for the 3 different things here: https://github.com/SAP/crossplane-provider-btp/pull/654/changes

I suggest we apply these changes to the other providers as well and document it here how it would look correctly after the change so we dont document an (to me)arbitrary hard-coding tht we can simply solve (and fall back to the hardcoded defaults if omitted).

enrico-kaack-comp · 2026-05-28T13:22:43Z

+- Small landscape ({'<' + '100'} resources): 1-5
+- Medium landscape (100-500): 5-10
+- Large landscape (500-2000): 10-20
+- XL landscape (2000+): 20+


Not that this will result in more API requests --> potencial API limits especially if they have a burst limit

Resolved in 7bdb028

enrico-kaack-comp · 2026-05-28T13:24:03Z

+**How it works:**
+- Tokens are replenished at N per second
+- Before reconciling, a token must be consumed
+- If no token available, reconcile is requeued after backoff


if the item was requeued before, it will be exectured without consuming token to prevent infinite requeue.
Important since having this small will result in many requeues that will than be allowed immediately and basically rendering this useless.

This contradicts to the sequential backoff. I have noted down this behaviour, we should double check how it actually goes.

Resolved in 213daa5

enrico-kaack-comp · 2026-05-28T13:25:56Z

+
+### 3. Poll Interval
+
+**What it controls:** How often external state is checked for drift (observe-only poll).


for a succesffully-reconciled resource that did not went into an error last reconcile

Resolved in d7379bf

enrico-kaack-comp · 2026-05-28T13:26:37Z

+**What it controls:** How often external state is checked for drift (observe-only poll).
+
+**Default:** 1 minute (most providers)
+


Mention the cross reference to PollJitter here too

Resolved in 257ad31

enrico-kaack-comp · 2026-05-28T13:28:23Z

+- Burst allows short spikes above QPS
+- Leader election renewal consumes 1 QPS per renewal cycle
+
+**Tuning:** If you increase MaxConcurrentReconciles, QPS/Burst scale automatically (N × 5). Do not set these manually unless you understand the call profile of your provider.


I would suggest something like: if you see that you are rate limited here, try to increase it while checking the api server if it can handle the load. Also maybe the kubeapi server itself has a limit per client that might be worth mentioning.

Resolved in 9fae87c

enrico-kaack-comp · 2026-05-28T14:17:02Z

+- Renewal is a KubeAPI call - consumes QPS
+- On failover: creates momentary thundering herd as all resources are requeued
+
+**Consideration:** For single-replica deployments, leader election is pure overhead. Some providers allow disabling it.


Add: the idea of multi-replica deployment is for immediate failover. Since we reconcile in poll interval only, a pod restart/rescheduling is not an issue and no high availability is necessary. Therefore it is not needed for most use cases.

Resolved in 7dba7a3

enrico-kaack-comp · 2026-05-28T14:59:58Z

+- Without jitter: all resources requeue exactly at the same time after a restart or failover
+- With jitter: requeues spread over PollInterval ± jitter%, reducing load spikes
+
+**Critical for:** Environments with many resources (>500) or frequent restarts.


should be enbled by all providers. No disadvantage but less worklaod on api on burst is a good thing. Espcailly if the controller is used by many teams it might be globally helpfull to have this jitter

Resolved in bfb9a37

enrico-kaack-comp · 2026-05-28T15:01:31Z

+**Default:** 1 minute (most providers); some use 3m, 5m, or 10m
+
+**How it works:**
+- If reconcile exceeds timeout, it's canceled with a context deadline error


It is also given a reconcileGracePeriod to clean up.

Resolved in 2335f1f

…netes-parameters Signed-off-by: Johannes Ott <johannes.ott@sap.com>

sk31337

@gergely-szabo-sap thanks for the gathered summary of important knowledge in one place to enable and up-skill our contributors.

Personally, i would suggest putting "Performance Considerations" Guide into a sub section for advanced topics or "maintaining your provider".

I would also welcome links of each section/headline to link to the original source/documentation.

Additionally, i am missing a more clear recommendation and instruction for contributors on what they should do/improve after reading the guide.

There is already the "Provider Defaults Comparison" but i miss the external links to the actual implementation for Provider BTP 3 MaxCons/RPS ect...

I very much like the "Motivation", "End-user Options" and "Contribution Options" which should be "Crossplane Provider Options" part of this PR.

Personally, I would take some big inspiration from this PR (but not replace your content with his 😉 )

sk31337 · 2026-06-15T09:13:53Z

+
+![Configuration Stack](/img/contribution/k8s-params/diagram-01-configuration-stack.svg)
+
+<!-- https://play.d2lang.com/?script=lJTfauM6EMbv_RQDuY3T_Gk5oIsDbWhDIQdykoVlITeqNLVFx1J2NG6TTf3ui2y33XZbSi8yxOMZzU8f81mcECp4rGwGMBgM4FuJMA_-1hU1a3HBw0a0ueveDuCKQwXz5TVckS4iSIDLvSB7TXC-uo7ZY5YZ0jFiVHDMAG7CvvsDEEu9QwWMRrQvCNvkg7NSKvhnPO5q5JBwuobUzRY5Z21dHRWc9ekoHO4w73u7ziZLvybL-hulI_qJgnvJADxqViBhlxv0gpy9GXcbvOTR_UIF027SAJxockaBcI1PA-hUwSNpwX0GsLUu7kgfyHmMR9guz9eLy-M2jTwu9QEZThWsONw7i_wiXNPAdgttmcjRVPZk19ecVNr5UREaaJKYaVjCa0VVSc4n7NGtI1KwdEUp88CaWrbZh2zvoM0UzDnEVOQxX9deXIWv0SrtdYF2tEYTvHGEPIQFhRtNay24dJWTlFoFousk6r2mpgOffQK-SZgXVGPLPf2SplMFJnjhQJTW4z1wSFvcVwzhe-C7nzXWOIT_9H4evKmZ0cvzteIQ5tqU2MNPP4FfBLLoOdgWfvIl-IkCQw695EV4y8wYZdTZbwj_rzYnFzVHGcISddqfS0KTPNlTTj6h_IFE4WHBiL7lHH-Jc6xee_sN6wb5HjmPziJQ2oM4hFKzBYOOnC-gaaClHH9E-fxhaO2sYI22fQ47bZwcFEyyv91-9mfO6lgqmL3Y1wQK_HRS71bI_wWatXHaxkkbx9nvAAAA__8%3D&layout=elk&-->


@gergely-szabo-sap can't this be done via mermaid plugin?

The content is not shown in markdown mode and a feel not confortable with such inclusion.

@gergely-szabo-sap can't this be done via mermaid plugin?

The content is not shown in markdown mode and a feel not confortable with such inclusion.

Resolved in eed9472

Co-authored-by: Enrico Kaack <enrico.kaack@sap.com>

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

gergely-szabo-sap · 2026-06-16T13:46:22Z

Personally, i would suggest putting "Performance Considerations" Guide into a sub section for advanced topics or "maintaining your provider".

Additionally, i am missing a more clear recommendation and instruction for contributors on what they should do/improve after reading the guide.

Thanks for the suggestion. I moved the guide out of the main contribution flow and placed it under a new Provider Operations section.

I slightly deviated from putting it under “Maintaining your provider” because the content is less about provider maintenance tasks like development, releases, or dependency updates, and more about operational concerns: runtime
behavior, reconciliation tuning, scaling, API throttling, timeouts, and failure modes.

Making Provider Operations a top-level section also keeps it discoverable for operators who may not consider themselves contributors, while still addressing the original concern that this guide should not sit directly in the basic
contribution guide sequence.

Resolved in 9500204

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

gergely-szabo-sap · 2026-06-17T12:35:48Z

I would also welcome links of each section/headline to link to the original source/documentation.

Resolved in b19ef3d

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

enrico-kaack-comp · 2026-06-17T14:59:38Z

Looks great.
Not in the loop of target audience etc., maybe have a on a quick glance on top with maybe poll intervall and suggested landscape sizes and the more in depth part below.

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

gergely-szabo-sap · 2026-06-18T08:47:03Z

Looks great.
Not in the loop of target audience etc., maybe have a on a quick glance on top with maybe poll intervall and suggested landscape sizes and the more in depth part below.

Resolved in c38395b

sdischer-sap · 2026-05-13T09:50:23Z

+
+### 1. MaxConcurrentReconciles
+
+**What it controls:** The number of worker goroutines that can reconcile resources in parallel.


per controller?

Resolved in 04de0db

sdischer-sap · 2026-06-22T13:22:08Z

+| **{'<' + '100'}** | 1-5 | 1m | Defaults are usually fine |
+| **100-500** | 5-10 | 1-5m | Add poll jitter if available |
+| **500-2000** | 10-20 | 3-5m | Add jitter and verify external API limits |
+| **2000+** | 20+ | 5-10m | External APIs usually become the bottleneck; consider sharding |


I guess those numbers are more or less randomly choosen right?
We do not yet have solid numbers to base on.
I guess this is what we can derive in a more concrete use case like for BTP in Q3 right (as part of https://github.tools.sap/cloud-orchestration/provider-stream-items/issues/150)?

Yes, the numbers are not very well-founded. We need more measurements for the solid numbers.

sdischer-sap · 2026-06-22T13:25:42Z

+
+## The --max-reconcile-rate Fan-Out
+
+> **Sources:** [Crossplane rate-limiting design][src-crossplane-rate-limiting], [crossplane-runtime rate limiter defaults][src-xruntime-ratelimiter], [BTP provider flags and wiring][src-btp-main], [Cloud Foundry provider flags and wiring][src-cf-main]


This document might be worth linking as well: https://sap.github.io/crossplane-provider-docs/docs/crossplane-provider-btp/docs/end-user-guides/setup/configure-controller-flags

Resolved in 6206634

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

gergely-szabo-sap · 2026-06-25T10:53:54Z

gergely-szabo-sap self-assigned this May 12, 2026

gergely-szabo-sap requested review from enrico-kaack-comp, jn-av and sdischer-sap May 12, 2026 13:50

gergely-szabo-sap force-pushed the feat/crossplane-kubernetes-parameters branch from 5e8aba3 to 2c5642a Compare May 18, 2026 07:47

enrico-kaack-comp requested changes May 28, 2026

View reviewed changes

GenosseOtt self-requested a review June 9, 2026 13:36

Merge remote-tracking branch 'origin/main' into feat/crossplane-kuber…

37235ea

…netes-parameters Signed-off-by: Johannes Ott <johannes.ott@sap.com>

GenosseOtt requested a review from a team as a code owner June 10, 2026 13:25

sk31337 suggested changes Jun 15, 2026

View reviewed changes

gergely-szabo-sap and others added 11 commits June 15, 2026 13:36

Update docs/contribution/7-understanding-k8s-crossplane-parameters.mdx

8237210

Co-authored-by: Enrico Kaack <enrico.kaack@sap.com>

Update docs/contribution/7-understanding-k8s-crossplane-parameters.mdx

c327a96

Co-authored-by: Enrico Kaack <enrico.kaack@sap.com>

docs: note API burst risk for reconcile concurrency

7bdb028

Clarify Crossplane global rate limiter behavior

213daa5

docs: clarify poll interval behavior

d7379bf

docs: reference poll jitter from poll interval

257ad31

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: clarify qps and burst tuning

9fae87c

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: clarify leader election consideration

7dba7a3

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: recommend enabling poll jitter by default

bfb9a37

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: clarify reconcile timeout grace period

2335f1f

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: move performance guide to provider operations

9500204

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: add source links to performance considerations

b19ef3d

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: replace performance diagrams with mermaid

eed9472

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

This comment was marked as resolved.

Sign in to view

gergely-szabo-sap requested review from enrico-kaack-comp and sk31337 June 17, 2026 14:03

docs: add quick tuning guide for provider performance

c38395b

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

This comment was marked as outdated.

Sign in to view

sdischer-sap reviewed Jun 22, 2026

View reviewed changes

gergely-szabo-sap added 2 commits June 25, 2026 10:38

docs: clarify reconcile concurrency scope

04de0db

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

docs: link BTP controller flags guide

6206634

Signed-off-by: gergely-szabo-sap <gergely.szabo@sap.com>

gergely-szabo-sap requested a review from sdischer-sap June 26, 2026 09:35


		### 3. Poll Interval

		What it controls: How often external state is checked for drift (observe-only poll).

		What it controls: How often external state is checked for drift (observe-only poll).

		Default: 1 minute (most providers)


		![Configuration Stack](/img/contribution/k8s-params/diagram-01-configuration-stack.svg)

		<!-- https://play.d2lang.com/?script=lJTfauM6EMbv_RQDuY3T_Gk5oIsDbWhDIQdykoVlITeqNLVFx1J2NG6TTf3ui2y33XZbSi8yxOMZzU8f81mcECp4rGwGMBgM4FuJMA_-1hU1a3HBw0a0ueveDuCKQwXz5TVckS4iSIDLvSB7TXC-uo7ZY5YZ0jFiVHDMAG7CvvsDEEu9QwWMRrQvCNvkg7NSKvhnPO5q5JBwuobUzRY5Z21dHRWc9ekoHO4w73u7ziZLvybL-hulI_qJgnvJADxqViBhlxv0gpy9GXcbvOTR_UIF027SAJxockaBcI1PA-hUwSNpwX0GsLUu7kgfyHmMR9guz9eLy-M2jTwu9QEZThWsONw7i_wiXNPAdgttmcjRVPZk19ecVNr5UREaaJKYaVjCa0VVSc4n7NGtI1KwdEUp88CaWrbZh2zvoM0UzDnEVOQxX9deXIWv0SrtdYF2tEYTvHGEPIQFhRtNay24dJWTlFoFousk6r2mpgOffQK-SZgXVGPLPf2SplMFJnjhQJTW4z1wSFvcVwzhe-C7nzXWOIT_9H4evKmZ0cvzteIQ5tqU2MNPP4FfBLLoOdgWfvIl-IkCQw695EV4y8wYZdTZbwj_rzYnFzVHGcISddqfS0KTPNlTTj6h_IFE4WHBiL7lHH-Jc6xee_sN6wb5HjmPziJQ2oM4hFKzBYOOnC-gaaClHH9E-fxhaO2sYI22fQ47bZwcFEyyv91-9mfO6lgqmL3Y1wQK_HRS71bI_wWatXHaxkkbx9nvAAAA__8%3D&layout=elk&-->


		### 1. MaxConcurrentReconciles

		What it controls: The number of worker goroutines that can reconcile resources in parallel.


		## The --max-reconcile-rate Fan-Out

		> Sources: [Crossplane rate-limiting design][src-crossplane-rate-limiting], [crossplane-runtime rate limiter defaults][src-xruntime-ratelimiter], [BTP provider flags and wiring][src-btp-main], [Cloud Foundry provider flags and wiring][src-cf-main]

Uh oh!

Conversation

gergely-szabo-sap commented May 12, 2026

Uh oh!

enrico-kaack-comp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sk31337 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gergely-szabo-sap commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gergely-szabo-sap commented Jun 17, 2026

Uh oh!

This comment was marked as resolved.

enrico-kaack-comp commented Jun 17, 2026

Uh oh!

gergely-szabo-sap commented Jun 18, 2026

Uh oh!

This comment was marked as outdated.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gergely-szabo-sap commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

gergely-szabo-sap commented Jun 16, 2026 •

edited

Loading