Feat/crossplane tuning knobs guide

docs/contribution/7-understanding-k8s-crossplane-parameters.mdx

@@ -0,0 +1,161 @@
+---
+sidebar_position: 7
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import Link from '@docusaurus/Link';
+
+# Performance Considerations
+
+## Motivation
+
+You deployed a Crossplane provider and **everything works — until you scale**. Resources start **reconciling slowly, rate limit errors appear**, and you're not sure which knob to turn.
+
+Throughput is bounded by the most restrictive layer in the stack — the external API is the ceiling, CLI flags are what you control.
+
+We investigated this in the context of our SAP providers — [crossplane-provider-btp](../crossplane-provider-btp/docs/end-user-guides/btp) and [crossplane-provider-cloudfoundry](../crossplane-provider-cloudfoundry/docs/end-user-guides/cloudfoundry) — but the concepts apply to any Crossplane provider.
+
+Depending on your role, jump to the relevant section:
+- **[End-user Options](#end-user-options)** — tune your deployment via CLI flags
+- **[Contribution Options](#contribution-options)** — change defaults or expose new flags in provider code
+
+:::info
+Scaling the external API itself is not an option in most cases and is therefore out of scope — it is the ceiling you tune against.
+:::
+
+## End-user Options
+
+As an operator deploying a Crossplane provider, **you can tune the following flags on the provider's `Deployment` or via a `DeploymentRuntimeConfig`**:
+
+Everything else — timeouts, backoff, jitter, leader election — is fixed at the values set by the provider developer. If defaults don't work for your scale, see [Contribution Options](#contribution-options).
+
+| Flag | What it changes | When to touch it |
+|------|----------------|-----------------|
+| `--max-reconcile-rate` | Concurrency, RPS, and KubeAPI QPS/Burst all at once | Your primary scaling lever — start here |
+| `--poll-interval` | How often drift is detected | Raise it when your landscape grows beyond 500 resources |
+
+
+### `--max-reconcile-rate`
+
+**What it controls:** A single flag that wires three coupled parameters at once:
+- **MaxConcurrentReconciles** — number of parallel worker goroutines
+- **GlobalRateLimiter (RPS)** — token-bucket rate, set to the same value as N
+- **QPS/Burst (client-go)** — HTTP rate to Kubernetes API, set to N×5 / N×10
+
+> **Note:** The coupling of these three concerns into one flag is a known design issue. A proposal to split them into separate flags exists in [crossplane-provider-btp#654](https://github.com/SAP/crossplane-provider-btp/pull/654).
+
+**Default:** Provider-specific ([BTP: 3](https://github.com/SAP/crossplane-provider-btp/blob/main/cmd/provider/main.go#L50), [CF: 10](https://github.com/SAP/crossplane-provider-cloudfoundry/blob/main/cmd/provider/main.go#L42), [HANA: 10](https://github.com/SAP/crossplane-provider-hana/blob/main/cmd/provider/main.go#L43), others: 100)
+
+<details>
+<summary>How it works</summary>
+
+- Each controller spawns exactly N workers (fixed count, not elastic)
+- A token-bucket rate limiter (RPS = N) gates every reconcile — if no token is available, the item is requeued. Exception: if an item was already requeued before, it executes without consuming a token to prevent infinite requeue loops
+- Each reconcile makes ~5–10 KubeAPI calls; QPS is set to N×5 and Burst to N×10 automatically
+- Bottleneck is typically I/O (network) wait, not CPU — concurrency can safely exceed CPU cores
+
+<Tabs>
+  <TabItem value="upjet" label="upjet/Terraform">
+    For Terraform-based controllers, concurrency > 1 may cause issues because Terraform itself is not thread-safe. Most upjet controllers set this to 1.
+  </TabItem>
+  <TabItem value="provider-sdk" label="Provider SDK (go-sdk)">
+    Go-based providers typically use concurrency > 1 safely. Default in provider-gcp, provider-helm, provider-kubernetes is 100.
+  </TabItem>
+</Tabs>
+
+</details>
+
+:::tip Tuning
+- **Small** landscape (< 100 resources): 1–5
+- **Medium** landscape (100–500): 5–10
+- **Large** landscape (500–2000): 10–20
+- **XL** landscape (2000+): 20+
+
+Higher concurrency increases the number of concurrent API requests to both external and Kubernetes APIs. Watch for rate limit errors when raising this value — especially burst limits on the KubeAPI server, which enforces per-client constraints.
+:::
+
+### `--poll-interval`
+
+**What it controls:** How often a successfully-reconciled resource (not in error state) is re-checked for drift.
+
+**Default:** 1 minute (most providers)
+
+<details>
+<summary>How it works</summary>
+
+- Creates steady-state RPS = N_resources / PollInterval
+- 1000 resources with 1m poll = ~16.7 polls/sec
+- Polls consume tokens from the global rate limiter
+
+</details>
+
+:::tip Tuning
+- **Small** (< 100): 1 minute
+- **Medium** (100–500): 1–5 minutes
+- **Large** (500–2000): 3–5 minutes
+- **XL** (2000+): 5–10 minutes
+:::
+
+> **Jitter note:** BTP and CF providers have no built-in jitter. Without jitter, all resources requeue simultaneously after a restart, creating a load spike. There is no disadvantage to enabling jitter — it only reduces burst load, which is especially beneficial when controllers are shared across teams. See [Contribution Options](#contribution-options) to add jitter support.
+
+### Tuning by Landscape Size
+
+| Resources | --max-reconcile-rate | --poll-interval | QPS / Burst (auto) | Notes |
+|-----------|---------------------|-----------------|---------------------|-------|
+| **< 100** | 1–5 | 1m | 25 / 50 | Defaults are fine |
+| **100–500** | 5–10 | 1–5m | 50 / 100 | CF default (10) reasonable |
+| **500–2000** | 10–20 | 3–5m | 150 / 300 | Must stay below external API limits |
+| **2000+** | 20+ | 5–10m | 250 / 500 | External APIs bottleneck; consider sharding |
+
+**Key principles:**
+1. External APIs are the **hard ceiling** — tune internal parameters below this limit
+2. Polling creates steady-state RPS = N ÷ P — factor this into your budget
+3. Raising `--max-reconcile-rate` raises concurrency, RPS, and KubeAPI load all at once
+
+---
+
+## Contribution Options
+
+As a provider contributor, you can change or expose the following that end users currently cannot touch:
+
+| What | How | Impact |
+|------|-----|--------|
+| **Decouple `--max-reconcile-rate`** | Split into separate flags for concurrency, RPS, and QPS (see [BTP#654](https://github.com/SAP/crossplane-provider-btp/pull/654)) | Lets operators tune each dimension independently |
+| **Change default `--max-reconcile-rate`** | Adjust the default in the provider's `main.go` | Affects all deployments that don't override the flag |
+| **Change default `--poll-interval`** | Adjust the default in the provider's `main.go` | Reduces steady-state API load for existing deployments |
+| **Add poll jitter** | Implement `PollJitterPercent` in the controller setup | No downside — reduces burst load, especially on shared controllers |
+| **Expose reconcile timeout as a flag** | Replace hardcoded `time.Minute` with a CLI flag | Lets operators handle slow external APIs without a code change |
+| **Change backoff values** | Override `RateLimiterWithContext` in controller setup | Useful when transient errors cause excessive 60s waits |
+| **Disable/tune leader election** | Pass `LeaderElection: false` or adjust lease durations | HA is unnecessary for most use cases — reconciliation runs on poll intervals, pod restarts are not critical |
+| **Adjust QPS/Burst multipliers** | Change the N×5 / N×10 formula in client config | Raise QPS if rate-limited by KubeAPI, but monitor API server load — it enforces per-client limits |
+
+---
+
+### Provider Defaults Comparison
+
+| Provider | MaxConc / RPS | Poll | Timeout | Jitter |
+|----------|---------------|------|---------|--------|
+| **BTP** | 3 | 1m | 1m (some 3m) | None |
+| **CF** | 10 | 1m | 1m (SI: 5m) | None |
+| provider-gcp | 100 | 10m | 3m | 5% (hardcoded) |
+| provider-helm | 100 | 10m | 10m | None |
+| provider-kubernetes | 100 | 10m | 1m | 10% (flag) |
+| provider-sql | 10 | 10m | 1m | None |
+
+---
+
+### Common Pitfalls
+
+1. **Ignoring external API limits** — Setting internal RPS higher than external API capacity generates 429 errors and spurious backoff
+2. **No poll jitter on large landscapes** — All resources requeue simultaneously after restart; raise poll interval to compensate or add jitter
+3. **Coupled concurrency and RPS** — BTP (upjet) needs MaxConc=1; go-sdk providers can safely run higher
+4. **Ignoring 429 handling** — Neither BTP nor CF providers parse rate-limit headers; 429s trigger generic backoff
+5. **Underestimating KubeAPI calls** — Each reconcile makes ~5–10 KubeAPI calls; QPS must be 5×–10× RPS
+6. **Setting GlobalRateLimiter too low** — Items that were previously requeued bypass the token check; a very low RPS effectively becomes useless as requeued items accumulate
+
+---
+
+## Further Reading
+
+- **Provider-specific tuning:** See individual provider documentation for provider-specific knobs and recommendations
+- **Metrics:** Monitor `workqueue_*` and `reconcile_*` metrics to understand actual throughput