From 12c5df61e1e3dc4779f605cfd770c0ad11ddda1d Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 16:49:29 +0200 Subject: [PATCH 1/8] docs(contribution): add Crossplane provider performance tuning guide Covers the two tunable CLI flags (--max-reconcile-rate, --poll-interval) with landscape-size guidance, contribution opportunities, and provider defaults for BTP, CF, and HANA. Structured for both end users and provider contributors. Signed-off-by: Johannes Ott --- ...nderstanding-k8s-crossplane-parameters.mdx | 155 ++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 docs/contribution/7-understanding-k8s-crossplane-parameters.mdx diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx new file mode 100644 index 0000000..c1d3866 --- /dev/null +++ b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx @@ -0,0 +1,155 @@ +--- +sidebar_position: 7 +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Link from '@docusaurus/Link'; + +# Performance Considerations + +## Motivation + +You deployed a Crossplane provider and **everything works — until you scale**. Resources start **reconciling slowly, rate limit errors appear**, and you're not sure which knob to turn. + +Throughput is bounded by the most restrictive layer in the stack — the external API is the ceiling, CLI flags are what you control. + +We investigated this in the context of our SAP providers — [crossplane-provider-btp](../crossplane-provider-btp/docs/end-user-guides/btp) and [crossplane-provider-cloudfoundry](../crossplane-provider-cloudfoundry/docs/end-user-guides/cloudfoundry) — but the concepts apply to any Crossplane provider. + +Depending on your role, jump to the relevant section: +- **[End-user Options](#end-user-options)** — tune your deployment via CLI flags +- **[Contribution Possibilities](#contribution-possibilities)** — change defaults or expose new flags in provider code + +:::info +Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against. +::: + +## End-user Options + +As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**: + +Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Possibilities](#contribution-possibilities). + +| Flag | What it changes | When to touch it | +|------|----------------|-----------------| +| `--max-reconcile-rate` | Concurrency, RPS, and KubeAPI QPS/Burst all at once | Your primary scaling lever — start here | +| `--poll-interval` | How often drift is detected | Raise it when your landscape grows beyond 500 resources | + + +### `--max-reconcile-rate` + +**What it controls:** A single flag that wires three coupled parameters at once: +- **MaxConcurrentReconciles** — number of parallel worker goroutines +- **GlobalRateLimiter (RPS)** — token-bucket rate, set to the same value as N +- **QPS/Burst (client-go)** — HTTP rate to Kubernetes API, set to N×5 / N×10 + +**Default:** Provider-specific ([BTP: 3](https://github.com/SAP/crossplane-provider-btp/blob/main/cmd/provider/main.go#L50), [CF: 10](https://github.com/SAP/crossplane-provider-cloudfoundry/blob/main/cmd/provider/main.go#L42), [HANA: 10](https://github.com/SAP/crossplane-provider-hana/blob/main/cmd/provider/main.go#L43), others: 100) + +
+How it works + +- Each controller spawns exactly N workers (fixed count, not elastic) +- A token-bucket rate limiter (RPS = N) gates every reconcile — if no token is available, the item is requeued +- Each reconcile makes ~5–10 KubeAPI calls; QPS is set to N×5 and Burst to N×10 automatically +- Bottleneck is typically I/O wait, not CPU — concurrency can safely exceed CPU cores + + + + For Terraform-based controllers, concurrency > 1 may cause issues because Terraform itself is not thread-safe. Most upjet controllers set this to 1. + + + Go-based providers typically use concurrency > 1 safely. Default in provider-gcp, provider-helm, provider-kubernetes is 100. + + + +
+ +:::tip Tuning +- **Small** landscape ({'<' + '100'} resources): 1–5 +- **Medium** landscape (100–500): 5–10 +- **Large** landscape (500–2000): 10–20 +- **XL** landscape (2000+): 20+ +::: + +### `--poll-interval` + +**What it controls:** How often external state is checked for drift. + +**Default:** 1 minute (most providers) + +
+How it works + +- Creates steady-state RPS = N_resources / PollInterval +- 1000 resources with 1m poll = ~16.7 polls/sec +- Polls consume tokens from the global rate limiter + +
+ +:::tip Tuning +- **Small** ({'<' + '100'}): 1 minute +- **Medium** (100–500): 1–5 minutes +- **Large** (500–2000): 3–5 minutes +- **XL** (2000+): 5–10 minutes +::: + +> **Jitter note:** BTP and CF providers have no built-in jitter. Without jitter, all resources requeue simultaneously after a restart. For large landscapes (500+), consider raising the poll interval to spread the load naturally, or open an issue to request jitter support. + +### Tuning by Landscape Size + +| Resources | --max-reconcile-rate | --poll-interval | QPS / Burst (auto) | Notes | +|-----------|---------------------|-----------------|---------------------|-------| +| **{'<' + '100'}** | 1–5 | 1m | 25 / 50 | Defaults are fine | +| **100–500** | 5–10 | 1–5m | 50 / 100 | CF default (10) reasonable | +| **500–2000** | 10–20 | 3–5m | 150 / 300 | Must stay below external API limits | +| **2000+** | 20+ | 5–10m | 250 / 500 | External APIs bottleneck; consider sharding | + +**Key principles:** +1. External APIs are the **hard ceiling** — tune internal parameters below this limit +2. Polling creates steady-state RPS = N ÷ P — factor this into your budget +3. Raising `--max-reconcile-rate` raises concurrency, RPS, and KubeAPI load all at once + +--- + +## Contribution Options + +As a provider contributor, you can change or expose the following that end users currently cannot touch: + +| What | How | Impact | +|------|-----|--------| +| **Change default `--max-reconcile-rate`** | Adjust the default in the provider's `main.go` | Affects all deployments that don't override the flag | +| **Change default `--poll-interval`** | Adjust the default in the provider's `main.go` | Reduces steady-state API load for existing deployments | +| **Add poll jitter** | Implement `PollJitterPercent` in the controller setup | Critical for BTP and CF on large landscapes | +| **Expose reconcile timeout as a flag** | Replace hardcoded `time.Minute` with a CLI flag | Lets operators handle slow external APIs without a code change | +| **Change backoff values** | Override `RateLimiterWithContext` in controller setup | Useful when transient errors cause excessive 60s waits | +| **Disable/tune leader election** | Pass `LeaderElection: false` or adjust lease durations | Reduces KubeAPI overhead for single-replica deployments | +| **Adjust QPS/Burst multipliers** | Change the N×5 / N×10 formula in client config | Useful if a provider's reconcile call profile differs significantly | + +--- + +### Provider Defaults Comparison + +| Provider | MaxConc / RPS | Poll | Timeout | Jitter | +|----------|---------------|------|---------|--------| +| **BTP** | 3 | 1m | 1m (some 3m) | None | +| **CF** | 10 | 1m | 1m (SI: 5m) | None | +| provider-gcp | 100 | 10m | 3m | 5% (hardcoded) | +| provider-helm | 100 | 10m | 10m | None | +| provider-kubernetes | 100 | 10m | 1m | 10% (flag) | +| provider-sql | 10 | 10m | 1m | None | + +--- + +### Common Pitfalls + +1. **Ignoring external API limits** — Setting internal RPS higher than external API capacity generates 429 errors and spurious backoff +2. **No poll jitter on large landscapes** — All resources requeue simultaneously after restart; raise poll interval to compensate +3. **Coupled concurrency and RPS** — BTP (upjet) needs MaxConc=1; go-sdk providers can safely run higher +4. **Ignoring 429 handling** — Neither BTP nor CF providers parse rate-limit headers; 429s trigger generic backoff +5. **Underestimating KubeAPI calls** — Each reconcile makes ~5–10 KubeAPI calls; QPS must be 5×–10× RPS + +--- + +## Further Reading + +- **Provider-specific tuning:** See individual provider documentation for provider-specific knobs and recommendations +- **Metrics:** Monitor `workqueue_*` and `reconcile_*` metrics to understand actual throughput From f2fde4555a00cfed8f80a5e86ca808688e01135a Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 16:50:11 +0200 Subject: [PATCH 2/8] fix: update broken anchor to renamed Contribution Options section Signed-off-by: Johannes Ott --- .../7-understanding-k8s-crossplane-parameters.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx index c1d3866..094ef8c 100644 --- a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx +++ b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx @@ -17,7 +17,7 @@ We investigated this in the context of our SAP providers — [crossplane-provide Depending on your role, jump to the relevant section: - **[End-user Options](#end-user-options)** — tune your deployment via CLI flags -- **[Contribution Possibilities](#contribution-possibilities)** — change defaults or expose new flags in provider code +- **[Contribution Options](#contribution-options)** — change defaults or expose new flags in provider code :::info Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against. @@ -27,7 +27,7 @@ Scaling the external API itself is not an option in most cases and is therefore As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**: -Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Possibilities](#contribution-possibilities). +Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Options](#contribution-options). | Flag | What it changes | When to touch it | |------|----------------|-----------------| From 86d097ba7bb63fe6dd2448740ad558fc31718f1d Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 16:51:48 +0200 Subject: [PATCH 3/8] Revert "fix: update broken anchor to renamed Contribution Options section" This reverts commit f2fde4555a00cfed8f80a5e86ca808688e01135a. Signed-off-by: Johannes Ott --- .../7-understanding-k8s-crossplane-parameters.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx index 094ef8c..c1d3866 100644 --- a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx +++ b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx @@ -17,7 +17,7 @@ We investigated this in the context of our SAP providers — [crossplane-provide Depending on your role, jump to the relevant section: - **[End-user Options](#end-user-options)** — tune your deployment via CLI flags -- **[Contribution Options](#contribution-options)** — change defaults or expose new flags in provider code +- **[Contribution Possibilities](#contribution-possibilities)** — change defaults or expose new flags in provider code :::info Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against. @@ -27,7 +27,7 @@ Scaling the external API itself is not an option in most cases and is therefore As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**: -Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Options](#contribution-options). +Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Possibilities](#contribution-possibilities). | Flag | What it changes | When to touch it | |------|----------------|-----------------| From 517f268ef4fc276008b02c59b69463f7f8fb63e1 Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 16:52:07 +0200 Subject: [PATCH 4/8] fix: correct broken anchor after section rename to Contribution Options Signed-off-by: Johannes Ott --- .../7-understanding-k8s-crossplane-parameters.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx index c1d3866..094ef8c 100644 --- a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx +++ b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx @@ -17,7 +17,7 @@ We investigated this in the context of our SAP providers — [crossplane-provide Depending on your role, jump to the relevant section: - **[End-user Options](#end-user-options)** — tune your deployment via CLI flags -- **[Contribution Possibilities](#contribution-possibilities)** — change defaults or expose new flags in provider code +- **[Contribution Options](#contribution-options)** — change defaults or expose new flags in provider code :::info Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against. @@ -27,7 +27,7 @@ Scaling the external API itself is not an option in most cases and is therefore As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**: -Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Possibilities](#contribution-possibilities). +Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Options](#contribution-options). | Flag | What it changes | When to touch it | |------|----------------|-----------------| From f764c1630790be279e470a230ca344b2268d0ce4 Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 16:57:55 +0200 Subject: [PATCH 5/8] Revert "docs(contribution): add Crossplane provider performance tuning guide" This reverts commit 12c5df61e1e3dc4779f605cfd770c0ad11ddda1d. Signed-off-by: Johannes Ott --- ...nderstanding-k8s-crossplane-parameters.mdx | 155 ------------------ 1 file changed, 155 deletions(-) delete mode 100644 docs/contribution/7-understanding-k8s-crossplane-parameters.mdx diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx deleted file mode 100644 index 094ef8c..0000000 --- a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx +++ /dev/null @@ -1,155 +0,0 @@ ---- -sidebar_position: 7 ---- -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import Link from '@docusaurus/Link'; - -# Performance Considerations - -## Motivation - -You deployed a Crossplane provider and **everything works — until you scale**. Resources start **reconciling slowly, rate limit errors appear**, and you're not sure which knob to turn. - -Throughput is bounded by the most restrictive layer in the stack — the external API is the ceiling, CLI flags are what you control. - -We investigated this in the context of our SAP providers — [crossplane-provider-btp](../crossplane-provider-btp/docs/end-user-guides/btp) and [crossplane-provider-cloudfoundry](../crossplane-provider-cloudfoundry/docs/end-user-guides/cloudfoundry) — but the concepts apply to any Crossplane provider. - -Depending on your role, jump to the relevant section: -- **[End-user Options](#end-user-options)** — tune your deployment via CLI flags -- **[Contribution Options](#contribution-options)** — change defaults or expose new flags in provider code - -:::info -Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against. -::: - -## End-user Options - -As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**: - -Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Options](#contribution-options). - -| Flag | What it changes | When to touch it | -|------|----------------|-----------------| -| `--max-reconcile-rate` | Concurrency, RPS, and KubeAPI QPS/Burst all at once | Your primary scaling lever — start here | -| `--poll-interval` | How often drift is detected | Raise it when your landscape grows beyond 500 resources | - - -### `--max-reconcile-rate` - -**What it controls:** A single flag that wires three coupled parameters at once: -- **MaxConcurrentReconciles** — number of parallel worker goroutines -- **GlobalRateLimiter (RPS)** — token-bucket rate, set to the same value as N -- **QPS/Burst (client-go)** — HTTP rate to Kubernetes API, set to N×5 / N×10 - -**Default:** Provider-specific ([BTP: 3](https://github.com/SAP/crossplane-provider-btp/blob/main/cmd/provider/main.go#L50), [CF: 10](https://github.com/SAP/crossplane-provider-cloudfoundry/blob/main/cmd/provider/main.go#L42), [HANA: 10](https://github.com/SAP/crossplane-provider-hana/blob/main/cmd/provider/main.go#L43), others: 100) - -
-How it works - -- Each controller spawns exactly N workers (fixed count, not elastic) -- A token-bucket rate limiter (RPS = N) gates every reconcile — if no token is available, the item is requeued -- Each reconcile makes ~5–10 KubeAPI calls; QPS is set to N×5 and Burst to N×10 automatically -- Bottleneck is typically I/O wait, not CPU — concurrency can safely exceed CPU cores - - - - For Terraform-based controllers, concurrency > 1 may cause issues because Terraform itself is not thread-safe. Most upjet controllers set this to 1. - - - Go-based providers typically use concurrency > 1 safely. Default in provider-gcp, provider-helm, provider-kubernetes is 100. - - - -
- -:::tip Tuning -- **Small** landscape ({'<' + '100'} resources): 1–5 -- **Medium** landscape (100–500): 5–10 -- **Large** landscape (500–2000): 10–20 -- **XL** landscape (2000+): 20+ -::: - -### `--poll-interval` - -**What it controls:** How often external state is checked for drift. - -**Default:** 1 minute (most providers) - -
-How it works - -- Creates steady-state RPS = N_resources / PollInterval -- 1000 resources with 1m poll = ~16.7 polls/sec -- Polls consume tokens from the global rate limiter - -
- -:::tip Tuning -- **Small** ({'<' + '100'}): 1 minute -- **Medium** (100–500): 1–5 minutes -- **Large** (500–2000): 3–5 minutes -- **XL** (2000+): 5–10 minutes -::: - -> **Jitter note:** BTP and CF providers have no built-in jitter. Without jitter, all resources requeue simultaneously after a restart. For large landscapes (500+), consider raising the poll interval to spread the load naturally, or open an issue to request jitter support. - -### Tuning by Landscape Size - -| Resources | --max-reconcile-rate | --poll-interval | QPS / Burst (auto) | Notes | -|-----------|---------------------|-----------------|---------------------|-------| -| **{'<' + '100'}** | 1–5 | 1m | 25 / 50 | Defaults are fine | -| **100–500** | 5–10 | 1–5m | 50 / 100 | CF default (10) reasonable | -| **500–2000** | 10–20 | 3–5m | 150 / 300 | Must stay below external API limits | -| **2000+** | 20+ | 5–10m | 250 / 500 | External APIs bottleneck; consider sharding | - -**Key principles:** -1. External APIs are the **hard ceiling** — tune internal parameters below this limit -2. Polling creates steady-state RPS = N ÷ P — factor this into your budget -3. Raising `--max-reconcile-rate` raises concurrency, RPS, and KubeAPI load all at once - ---- - -## Contribution Options - -As a provider contributor, you can change or expose the following that end users currently cannot touch: - -| What | How | Impact | -|------|-----|--------| -| **Change default `--max-reconcile-rate`** | Adjust the default in the provider's `main.go` | Affects all deployments that don't override the flag | -| **Change default `--poll-interval`** | Adjust the default in the provider's `main.go` | Reduces steady-state API load for existing deployments | -| **Add poll jitter** | Implement `PollJitterPercent` in the controller setup | Critical for BTP and CF on large landscapes | -| **Expose reconcile timeout as a flag** | Replace hardcoded `time.Minute` with a CLI flag | Lets operators handle slow external APIs without a code change | -| **Change backoff values** | Override `RateLimiterWithContext` in controller setup | Useful when transient errors cause excessive 60s waits | -| **Disable/tune leader election** | Pass `LeaderElection: false` or adjust lease durations | Reduces KubeAPI overhead for single-replica deployments | -| **Adjust QPS/Burst multipliers** | Change the N×5 / N×10 formula in client config | Useful if a provider's reconcile call profile differs significantly | - ---- - -### Provider Defaults Comparison - -| Provider | MaxConc / RPS | Poll | Timeout | Jitter | -|----------|---------------|------|---------|--------| -| **BTP** | 3 | 1m | 1m (some 3m) | None | -| **CF** | 10 | 1m | 1m (SI: 5m) | None | -| provider-gcp | 100 | 10m | 3m | 5% (hardcoded) | -| provider-helm | 100 | 10m | 10m | None | -| provider-kubernetes | 100 | 10m | 1m | 10% (flag) | -| provider-sql | 10 | 10m | 1m | None | - ---- - -### Common Pitfalls - -1. **Ignoring external API limits** — Setting internal RPS higher than external API capacity generates 429 errors and spurious backoff -2. **No poll jitter on large landscapes** — All resources requeue simultaneously after restart; raise poll interval to compensate -3. **Coupled concurrency and RPS** — BTP (upjet) needs MaxConc=1; go-sdk providers can safely run higher -4. **Ignoring 429 handling** — Neither BTP nor CF providers parse rate-limit headers; 429s trigger generic backoff -5. **Underestimating KubeAPI calls** — Each reconcile makes ~5–10 KubeAPI calls; QPS must be 5×–10× RPS - ---- - -## Further Reading - -- **Provider-specific tuning:** See individual provider documentation for provider-specific knobs and recommendations -- **Metrics:** Monitor `workqueue_*` and `reconcile_*` metrics to understand actual throughput From b7fb82f37e360b6bd15111da3d2b42efe9626839 Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 16:49:29 +0200 Subject: [PATCH 6/8] docs(contribution): add Crossplane provider performance tuning guide Covers the two tunable CLI flags (--max-reconcile-rate, --poll-interval) with landscape-size guidance, contribution opportunities, and provider defaults for BTP, CF, and HANA. Structured for both end users and provider contributors. Signed-off-by: Johannes Ott --- ...nderstanding-k8s-crossplane-parameters.mdx | 155 ++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 docs/contribution/7-understanding-k8s-crossplane-parameters.mdx diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx new file mode 100644 index 0000000..c1d3866 --- /dev/null +++ b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx @@ -0,0 +1,155 @@ +--- +sidebar_position: 7 +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import Link from '@docusaurus/Link'; + +# Performance Considerations + +## Motivation + +You deployed a Crossplane provider and **everything works — until you scale**. Resources start **reconciling slowly, rate limit errors appear**, and you're not sure which knob to turn. + +Throughput is bounded by the most restrictive layer in the stack — the external API is the ceiling, CLI flags are what you control. + +We investigated this in the context of our SAP providers — [crossplane-provider-btp](../crossplane-provider-btp/docs/end-user-guides/btp) and [crossplane-provider-cloudfoundry](../crossplane-provider-cloudfoundry/docs/end-user-guides/cloudfoundry) — but the concepts apply to any Crossplane provider. + +Depending on your role, jump to the relevant section: +- **[End-user Options](#end-user-options)** — tune your deployment via CLI flags +- **[Contribution Possibilities](#contribution-possibilities)** — change defaults or expose new flags in provider code + +:::info +Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against. +::: + +## End-user Options + +As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**: + +Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Possibilities](#contribution-possibilities). + +| Flag | What it changes | When to touch it | +|------|----------------|-----------------| +| `--max-reconcile-rate` | Concurrency, RPS, and KubeAPI QPS/Burst all at once | Your primary scaling lever — start here | +| `--poll-interval` | How often drift is detected | Raise it when your landscape grows beyond 500 resources | + + +### `--max-reconcile-rate` + +**What it controls:** A single flag that wires three coupled parameters at once: +- **MaxConcurrentReconciles** — number of parallel worker goroutines +- **GlobalRateLimiter (RPS)** — token-bucket rate, set to the same value as N +- **QPS/Burst (client-go)** — HTTP rate to Kubernetes API, set to N×5 / N×10 + +**Default:** Provider-specific ([BTP: 3](https://github.com/SAP/crossplane-provider-btp/blob/main/cmd/provider/main.go#L50), [CF: 10](https://github.com/SAP/crossplane-provider-cloudfoundry/blob/main/cmd/provider/main.go#L42), [HANA: 10](https://github.com/SAP/crossplane-provider-hana/blob/main/cmd/provider/main.go#L43), others: 100) + +
+How it works + +- Each controller spawns exactly N workers (fixed count, not elastic) +- A token-bucket rate limiter (RPS = N) gates every reconcile — if no token is available, the item is requeued +- Each reconcile makes ~5–10 KubeAPI calls; QPS is set to N×5 and Burst to N×10 automatically +- Bottleneck is typically I/O wait, not CPU — concurrency can safely exceed CPU cores + + + + For Terraform-based controllers, concurrency > 1 may cause issues because Terraform itself is not thread-safe. Most upjet controllers set this to 1. + + + Go-based providers typically use concurrency > 1 safely. Default in provider-gcp, provider-helm, provider-kubernetes is 100. + + + +
+ +:::tip Tuning +- **Small** landscape ({'<' + '100'} resources): 1–5 +- **Medium** landscape (100–500): 5–10 +- **Large** landscape (500–2000): 10–20 +- **XL** landscape (2000+): 20+ +::: + +### `--poll-interval` + +**What it controls:** How often external state is checked for drift. + +**Default:** 1 minute (most providers) + +
+How it works + +- Creates steady-state RPS = N_resources / PollInterval +- 1000 resources with 1m poll = ~16.7 polls/sec +- Polls consume tokens from the global rate limiter + +
+ +:::tip Tuning +- **Small** ({'<' + '100'}): 1 minute +- **Medium** (100–500): 1–5 minutes +- **Large** (500–2000): 3–5 minutes +- **XL** (2000+): 5–10 minutes +::: + +> **Jitter note:** BTP and CF providers have no built-in jitter. Without jitter, all resources requeue simultaneously after a restart. For large landscapes (500+), consider raising the poll interval to spread the load naturally, or open an issue to request jitter support. + +### Tuning by Landscape Size + +| Resources | --max-reconcile-rate | --poll-interval | QPS / Burst (auto) | Notes | +|-----------|---------------------|-----------------|---------------------|-------| +| **{'<' + '100'}** | 1–5 | 1m | 25 / 50 | Defaults are fine | +| **100–500** | 5–10 | 1–5m | 50 / 100 | CF default (10) reasonable | +| **500–2000** | 10–20 | 3–5m | 150 / 300 | Must stay below external API limits | +| **2000+** | 20+ | 5–10m | 250 / 500 | External APIs bottleneck; consider sharding | + +**Key principles:** +1. External APIs are the **hard ceiling** — tune internal parameters below this limit +2. Polling creates steady-state RPS = N ÷ P — factor this into your budget +3. Raising `--max-reconcile-rate` raises concurrency, RPS, and KubeAPI load all at once + +--- + +## Contribution Options + +As a provider contributor, you can change or expose the following that end users currently cannot touch: + +| What | How | Impact | +|------|-----|--------| +| **Change default `--max-reconcile-rate`** | Adjust the default in the provider's `main.go` | Affects all deployments that don't override the flag | +| **Change default `--poll-interval`** | Adjust the default in the provider's `main.go` | Reduces steady-state API load for existing deployments | +| **Add poll jitter** | Implement `PollJitterPercent` in the controller setup | Critical for BTP and CF on large landscapes | +| **Expose reconcile timeout as a flag** | Replace hardcoded `time.Minute` with a CLI flag | Lets operators handle slow external APIs without a code change | +| **Change backoff values** | Override `RateLimiterWithContext` in controller setup | Useful when transient errors cause excessive 60s waits | +| **Disable/tune leader election** | Pass `LeaderElection: false` or adjust lease durations | Reduces KubeAPI overhead for single-replica deployments | +| **Adjust QPS/Burst multipliers** | Change the N×5 / N×10 formula in client config | Useful if a provider's reconcile call profile differs significantly | + +--- + +### Provider Defaults Comparison + +| Provider | MaxConc / RPS | Poll | Timeout | Jitter | +|----------|---------------|------|---------|--------| +| **BTP** | 3 | 1m | 1m (some 3m) | None | +| **CF** | 10 | 1m | 1m (SI: 5m) | None | +| provider-gcp | 100 | 10m | 3m | 5% (hardcoded) | +| provider-helm | 100 | 10m | 10m | None | +| provider-kubernetes | 100 | 10m | 1m | 10% (flag) | +| provider-sql | 10 | 10m | 1m | None | + +--- + +### Common Pitfalls + +1. **Ignoring external API limits** — Setting internal RPS higher than external API capacity generates 429 errors and spurious backoff +2. **No poll jitter on large landscapes** — All resources requeue simultaneously after restart; raise poll interval to compensate +3. **Coupled concurrency and RPS** — BTP (upjet) needs MaxConc=1; go-sdk providers can safely run higher +4. **Ignoring 429 handling** — Neither BTP nor CF providers parse rate-limit headers; 429s trigger generic backoff +5. **Underestimating KubeAPI calls** — Each reconcile makes ~5–10 KubeAPI calls; QPS must be 5×–10× RPS + +--- + +## Further Reading + +- **Provider-specific tuning:** See individual provider documentation for provider-specific knobs and recommendations +- **Metrics:** Monitor `workqueue_*` and `reconcile_*` metrics to understand actual throughput From 3c0a9f4418609dde530100724881d4bf1b4e59a5 Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 16:52:07 +0200 Subject: [PATCH 7/8] fix: correct broken anchor after section rename to Contribution Options Signed-off-by: Johannes Ott --- .../7-understanding-k8s-crossplane-parameters.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx index c1d3866..094ef8c 100644 --- a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx +++ b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx @@ -17,7 +17,7 @@ We investigated this in the context of our SAP providers — [crossplane-provide Depending on your role, jump to the relevant section: - **[End-user Options](#end-user-options)** — tune your deployment via CLI flags -- **[Contribution Possibilities](#contribution-possibilities)** — change defaults or expose new flags in provider code +- **[Contribution Options](#contribution-options)** — change defaults or expose new flags in provider code :::info Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against. @@ -27,7 +27,7 @@ Scaling the external API itself is not an option in most cases and is therefore As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**: -Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Possibilities](#contribution-possibilities). +Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Options](#contribution-options). | Flag | What it changes | When to touch it | |------|----------------|-----------------| From ea259ecc8502809126cc8ffc7183855d31385cc4 Mon Sep 17 00:00:00 2001 From: Johannes Ott Date: Wed, 10 Jun 2026 17:06:03 +0200 Subject: [PATCH 8/8] docs: incorporate Enrico's review feedback into tuning guide Signed-off-by: Johannes Ott --- ...nderstanding-k8s-crossplane-parameters.mdx | 28 +++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx index 094ef8c..ecebcdc 100644 --- a/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx +++ b/docs/contribution/7-understanding-k8s-crossplane-parameters.mdx @@ -42,15 +42,17 @@ Everything else — timeouts, backoff, jitter, leader election — is fixed at t - **GlobalRateLimiter (RPS)** — token-bucket rate, set to the same value as N - **QPS/Burst (client-go)** — HTTP rate to Kubernetes API, set to N×5 / N×10 +> **Note:** The coupling of these three concerns into one flag is a known design issue. A proposal to split them into separate flags exists in [crossplane-provider-btp#654](https://github.com/SAP/crossplane-provider-btp/pull/654). + **Default:** Provider-specific ([BTP: 3](https://github.com/SAP/crossplane-provider-btp/blob/main/cmd/provider/main.go#L50), [CF: 10](https://github.com/SAP/crossplane-provider-cloudfoundry/blob/main/cmd/provider/main.go#L42), [HANA: 10](https://github.com/SAP/crossplane-provider-hana/blob/main/cmd/provider/main.go#L43), others: 100)
How it works - Each controller spawns exactly N workers (fixed count, not elastic) -- A token-bucket rate limiter (RPS = N) gates every reconcile — if no token is available, the item is requeued +- A token-bucket rate limiter (RPS = N) gates every reconcile — if no token is available, the item is requeued. Exception: if an item was already requeued before, it executes without consuming a token to prevent infinite requeue loops - Each reconcile makes ~5–10 KubeAPI calls; QPS is set to N×5 and Burst to N×10 automatically -- Bottleneck is typically I/O wait, not CPU — concurrency can safely exceed CPU cores +- Bottleneck is typically I/O (network) wait, not CPU — concurrency can safely exceed CPU cores @@ -64,15 +66,17 @@ Everything else — timeouts, backoff, jitter, leader election — is fixed at t
:::tip Tuning -- **Small** landscape ({'<' + '100'} resources): 1–5 +- **Small** landscape (< 100 resources): 1–5 - **Medium** landscape (100–500): 5–10 - **Large** landscape (500–2000): 10–20 - **XL** landscape (2000+): 20+ + +Higher concurrency increases the number of concurrent API requests to both external and Kubernetes APIs. Watch for rate limit errors when raising this value — especially burst limits on the KubeAPI server, which enforces per-client constraints. ::: ### `--poll-interval` -**What it controls:** How often external state is checked for drift. +**What it controls:** How often a successfully-reconciled resource (not in error state) is re-checked for drift. **Default:** 1 minute (most providers) @@ -86,19 +90,19 @@ Everything else — timeouts, backoff, jitter, leader election — is fixed at t :::tip Tuning -- **Small** ({'<' + '100'}): 1 minute +- **Small** (< 100): 1 minute - **Medium** (100–500): 1–5 minutes - **Large** (500–2000): 3–5 minutes - **XL** (2000+): 5–10 minutes ::: -> **Jitter note:** BTP and CF providers have no built-in jitter. Without jitter, all resources requeue simultaneously after a restart. For large landscapes (500+), consider raising the poll interval to spread the load naturally, or open an issue to request jitter support. +> **Jitter note:** BTP and CF providers have no built-in jitter. Without jitter, all resources requeue simultaneously after a restart, creating a load spike. There is no disadvantage to enabling jitter — it only reduces burst load, which is especially beneficial when controllers are shared across teams. See [Contribution Options](#contribution-options) to add jitter support. ### Tuning by Landscape Size | Resources | --max-reconcile-rate | --poll-interval | QPS / Burst (auto) | Notes | |-----------|---------------------|-----------------|---------------------|-------| -| **{'<' + '100'}** | 1–5 | 1m | 25 / 50 | Defaults are fine | +| **< 100** | 1–5 | 1m | 25 / 50 | Defaults are fine | | **100–500** | 5–10 | 1–5m | 50 / 100 | CF default (10) reasonable | | **500–2000** | 10–20 | 3–5m | 150 / 300 | Must stay below external API limits | | **2000+** | 20+ | 5–10m | 250 / 500 | External APIs bottleneck; consider sharding | @@ -116,13 +120,14 @@ As a provider contributor, you can change or expose the following that end users | What | How | Impact | |------|-----|--------| +| **Decouple `--max-reconcile-rate`** | Split into separate flags for concurrency, RPS, and QPS (see [BTP#654](https://github.com/SAP/crossplane-provider-btp/pull/654)) | Lets operators tune each dimension independently | | **Change default `--max-reconcile-rate`** | Adjust the default in the provider's `main.go` | Affects all deployments that don't override the flag | | **Change default `--poll-interval`** | Adjust the default in the provider's `main.go` | Reduces steady-state API load for existing deployments | -| **Add poll jitter** | Implement `PollJitterPercent` in the controller setup | Critical for BTP and CF on large landscapes | +| **Add poll jitter** | Implement `PollJitterPercent` in the controller setup | No downside — reduces burst load, especially on shared controllers | | **Expose reconcile timeout as a flag** | Replace hardcoded `time.Minute` with a CLI flag | Lets operators handle slow external APIs without a code change | | **Change backoff values** | Override `RateLimiterWithContext` in controller setup | Useful when transient errors cause excessive 60s waits | -| **Disable/tune leader election** | Pass `LeaderElection: false` or adjust lease durations | Reduces KubeAPI overhead for single-replica deployments | -| **Adjust QPS/Burst multipliers** | Change the N×5 / N×10 formula in client config | Useful if a provider's reconcile call profile differs significantly | +| **Disable/tune leader election** | Pass `LeaderElection: false` or adjust lease durations | HA is unnecessary for most use cases — reconciliation runs on poll intervals, pod restarts are not critical | +| **Adjust QPS/Burst multipliers** | Change the N×5 / N×10 formula in client config | Raise QPS if rate-limited by KubeAPI, but monitor API server load — it enforces per-client limits | --- @@ -142,10 +147,11 @@ As a provider contributor, you can change or expose the following that end users ### Common Pitfalls 1. **Ignoring external API limits** — Setting internal RPS higher than external API capacity generates 429 errors and spurious backoff -2. **No poll jitter on large landscapes** — All resources requeue simultaneously after restart; raise poll interval to compensate +2. **No poll jitter on large landscapes** — All resources requeue simultaneously after restart; raise poll interval to compensate or add jitter 3. **Coupled concurrency and RPS** — BTP (upjet) needs MaxConc=1; go-sdk providers can safely run higher 4. **Ignoring 429 handling** — Neither BTP nor CF providers parse rate-limit headers; 429s trigger generic backoff 5. **Underestimating KubeAPI calls** — Each reconcile makes ~5–10 KubeAPI calls; QPS must be 5×–10× RPS +6. **Setting GlobalRateLimiter too low** — Items that were previously requeued bypass the token check; a very low RPS effectively becomes useless as requeued items accumulate ---