From 1496c5293219def088ddb4d6f9ec7f7b51b18559 Mon Sep 17 00:00:00 2001 From: Brandon Chavis Date: Tue, 6 Jan 2026 12:08:48 -0500 Subject: [PATCH 1/4] Update worker-versioning.mdx --- .../worker-deployments/worker-versioning.mdx | 298 +++++++++++++++++- 1 file changed, 297 insertions(+), 1 deletion(-) diff --git a/docs/production-deployment/worker-deployments/worker-versioning.mdx b/docs/production-deployment/worker-deployments/worker-versioning.mdx index 01e1e9db4c..e67e9963f0 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning.mdx @@ -216,7 +216,40 @@ worker = Temporalio::Worker.new( -### Which Default Versioning Behavior should you choose? + + +## Choosing a Versioning Behavior {#choosing-behavior} + +The right versioning behavior depends on how long your Workflows run relative to your deployment frequency. + +### Decision guide {#decision-guide} + +| Workflow Duration | Uses Continue-as-New? | Recommended Behavior | Patching Required? | +|-------------------|----------------------|---------------------|-------------------| +| **Short** (completes before next deploy) | N/A | `PINNED` | Never | +| **Medium** (spans multiple deploys) | No | `AUTO_UPGRADE` | Yes | +| **Long** (weeks to years) | Yes | `PINNED` + [upgrade on CaN](#upgrade-on-continue-as-new) | Never | +| **Long** (weeks to years) | No | `AUTO_UPGRADE` + patching | Yes | + +### Examples by Workflow type {#behavior-examples} + +| Workflow Type | Duration | Recommended Behavior | Notes | +|---------------|----------|---------------------|-------| +| Order processing | Minutes | `PINNED` | Completes before next deploy | +| Payment retry | Hours | `PINNED` or `AUTO_UPGRADE` | Depends on deploy frequency | +| Subscription billing | Days | `AUTO_UPGRADE` | May span multiple deploys | +| Customer entity | Months-Years | `PINNED` + upgrade on CaN | Uses Continue-as-New pattern | +| AI agent / Chatbot | Weeks | `PINNED` + upgrade on CaN | Long sleeps, uses CaN | +| Compliance audit | Months | `AUTO_UPGRADE` + patching | Cannot use CaN (needs full history) | + +:::info Long-running Workflows with Continue-as-New + +If your Workflow uses Continue-as-New to manage history size, you can upgrade to new Worker Deployment Versions at the CaN boundary without patching. +See [Upgrading on Continue-as-New](#upgrade-on-continue-as-new) below. + +::: + +### Default Versioning Behavior Considerations If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning. Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`. @@ -236,6 +269,7 @@ Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow You also want to make sure you understand how your Activities are going to work across different Worker Deployment Versions. Refer to the [Worker Versioning Activitiy behavior docs](/worker-versioning#actvity-behavior-across-versions) for more details. + ## Rolling out changes with the CLI Next, deploy your Worker with the additional configuration parameters. @@ -439,6 +473,268 @@ temporal workflow update-options \ When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the Workflow's Target Version is different from the earlier Pinned Version, you should make sure you [patch](/patching#patching) the Workflow code. ::: +## Upgrading on Continue-as-New {#upgrade-on-continue-as-new} + +Long-running Workflows that use [Continue-as-New](/workflow-execution/continue-as-new) can upgrade to newer Worker Deployment Versions at Continue-as-New boundaries without requiring patching. + +This pattern is ideal for: +- **Entity Workflows** that run for months or years +- **Batch processing** Workflows that checkpoint with Continue-as-New +- **AI agent Workflows** with long sleeps waiting for user input + +:::note Public Preview + +This feature is in Public Preview as an experimental SDK-level option. + +::: + +### How it works {#upgrade-on-can-how-it-works} + +By default, Pinned Workflows stay on their original Worker Deployment Version even when they Continue-as-New. +With the upgrade option enabled: + +1. Each Workflow run remains pinned to its version (no patching needed during a run) +2. The Temporal Server suggests Continue-as-New when a new version becomes available +3. When the Workflow performs Continue-as-New with the upgrade option, the new run starts on the Current or Ramping version + +### Checking for new versions {#checking-for-new-versions} + +When a new Worker Deployment Version becomes Current or Ramping, active Workflows can detect this through `continue_as_new_suggested`. +Check for the `TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED` reason: + + + + +```go +import ( + "go.temporal.io/sdk/workflow" +) + +func MyEntityWorkflow(ctx workflow.Context, state EntityState) error { + for { + // Your workflow logic here + + // Check if we should continue as new + if workflow.IsContinueAsNewSuggested(ctx) { + info := workflow.GetInfo(ctx) + for _, reason := range info.ContinueAsNewSuggestedReasons { + if reason == workflow.ContinueAsNewSuggestedReasonTargetVersionChanged { + // A new Worker Deployment Version is available + // Continue-as-New with upgrade to the new version + return workflow.NewContinueAsNewError( + ctx, + MyEntityWorkflow, + state, + workflow.WithContinueAsNewVersioningBehavior(workflow.VersioningBehaviorAutoUpgrade), + ) + } + } + // Other CaN reasons (history size) - continue without upgrading + return workflow.NewContinueAsNewError(ctx, MyEntityWorkflow, state) + } + + // Wait for signals, timers, etc. + selector := workflow.NewSelector(ctx) + selector.AddReceive(workflow.GetSignalChannel(ctx, "update"), func(c workflow.ReceiveChannel, more bool) { + // Handle signal + }) + selector.Select(ctx) + } +} +``` + + + + +```python +from datetime import timedelta +from temporalio import workflow +from temporalio.workflow import ContinueAsNewSuggestedReason, VersioningBehavior + +@workflow.defn +class MyEntityWorkflow: + @workflow.run + async def run(self, state: EntityState) -> None: + while True: + # Your workflow logic here + + # Check if we should continue as new + if workflow.continue_as_new_suggested(): + info = workflow.info() + for reason in info.continue_as_new_suggested_reasons: + if reason == ContinueAsNewSuggestedReason.TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED: + # A new Worker Deployment Version is available + # Continue-as-New with upgrade to the new version + workflow.continue_as_new( + args=[state], + versioning_behavior=VersioningBehavior.AUTO_UPGRADE, + ) + + # Other CaN reasons (history size) - continue without upgrading + workflow.continue_as_new(args=[state]) + + # Wait for signals, timers, etc. + await workflow.wait_condition( + lambda: self.has_pending_work or workflow.continue_as_new_suggested(), + timeout=timedelta(hours=1), + ) +``` + + + + +```typescript +import { + continueAsNew, + continueAsNewSuggested, + workflowInfo, + condition, + ContinueAsNewSuggestedReason, +} from '@temporalio/workflow'; + +export async function myEntityWorkflow(state: EntityState): Promise { + while (true) { + // Your workflow logic here + + // Check if we should continue as new + if (continueAsNewSuggested()) { + const info = workflowInfo(); + for (const reason of info.continueAsNewSuggestedReasons ?? []) { + if (reason === ContinueAsNewSuggestedReason.TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED) { + // A new Worker Deployment Version is available + // Continue-as-New with upgrade to the new version + await continueAsNew(state, { + versioningBehavior: 'AUTO_UPGRADE', + }); + } + } + // Other CaN reasons (history size) - continue without upgrading + await continueAsNew(state); + } + + // Wait for signals, timers, etc. + await condition(() => hasPendingWork || continueAsNewSuggested(), '1h'); + } +} +``` + + + + +```java +import io.temporal.workflow.Workflow; +import io.temporal.workflow.ContinueAsNewOptions; +import io.temporal.api.enums.v1.ContinueAsNewSuggestedReason; +import io.temporal.api.enums.v1.VersioningBehavior; + +public class MyEntityWorkflowImpl implements MyEntityWorkflow { + @Override + public void run(EntityState state) { + while (true) { + // Your workflow logic here + + // Check if we should continue as new + if (Workflow.isContinueAsNewSuggested()) { + var info = Workflow.getInfo(); + for (var reason : info.getContinueAsNewSuggestedReasons()) { + if (reason == ContinueAsNewSuggestedReason.TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED) { + // A new Worker Deployment Version is available + Workflow.continueAsNew( + ContinueAsNewOptions.newBuilder() + .setVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) + .build(), + state + ); + } + } + // Other CaN reasons - continue without upgrading + Workflow.continueAsNew(state); + } + + // Wait for signals, timers, etc. + Workflow.await(() -> hasPendingWork || Workflow.isContinueAsNewSuggested()); + } + } +} +``` + + + + +```csharp +using Temporalio.Workflows; + +[Workflow] +public class MyEntityWorkflow +{ + [WorkflowRun] + public async Task RunAsync(EntityState state) + { + while (true) + { + // Your workflow logic here + + // Check if we should continue as new + if (Workflow.ContinueAsNewSuggested) + { + var info = Workflow.Info; + foreach (var reason in info.ContinueAsNewSuggestedReasons) + { + if (reason == ContinueAsNewSuggestedReason.TargetWorkerDeploymentVersionChanged) + { + // A new Worker Deployment Version is available + throw Workflow.CreateContinueAsNewException( + wf => wf.RunAsync(state), + new() { VersioningBehavior = VersioningBehavior.AutoUpgrade }); + } + } + // Other CaN reasons - continue without upgrading + throw Workflow.CreateContinueAsNewException( + wf => wf.RunAsync(state)); + } + + // Wait for signals, timers, etc. + await Workflow.WaitConditionAsync( + () => hasPendingWork || Workflow.ContinueAsNewSuggested, + TimeSpan.FromHours(1)); + } + } +} +``` + + + + +### Continue-as-New suggested reasons {#can-suggested-reasons} + +The `continue_as_new_suggested` mechanism can return multiple reasons: + +| Reason | Description | Action | +|--------|-------------|--------| +| `TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED` | A new Worker Deployment Version is available | Use upgrade option when continuing | +| `EVENTS_HISTORY_SIZE` | History size approaching limit | Continue-as-New to reset history | +| `EVENTS_HISTORY_LENGTH` | History event count approaching limit | Continue-as-New to reset history | + +:::tip + +Handle different reasons differently. +For version upgrades, use the `AUTO_UPGRADE` versioning behavior option. +For history size reasons, you may want to continue without upgrading to stay on the same version. + +::: + +### Limitations {#upgrade-on-can-limitations} + +:::caution Current Limitations + +- **Lazy moving only:** Workflows must wake up naturally to receive the Continue-as-New suggestion. + Sleeping Workflows won't be proactively signaled (planned for a future release). +- **Interface compatibility:** When continuing as new to a different version, ensure your Workflow input format is compatible. + If incompatible, the new run may fail on its first Workflow Task. + +::: + + ## Sunsetting an old Deployment Version A Worker Deployment Version moves through the following states: From fc32b109ca91342654e452f86788cf183809896e Mon Sep 17 00:00:00 2001 From: Brandon Chavis Date: Tue, 6 Jan 2026 12:53:53 -0500 Subject: [PATCH 2/4] Update worker-versioning.mdx --- .../worker-deployments/worker-versioning.mdx | 7 ------- 1 file changed, 7 deletions(-) diff --git a/docs/production-deployment/worker-deployments/worker-versioning.mdx b/docs/production-deployment/worker-deployments/worker-versioning.mdx index e67e9963f0..744889d3e5 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning.mdx @@ -715,13 +715,6 @@ The `continue_as_new_suggested` mechanism can return multiple reasons: | `EVENTS_HISTORY_SIZE` | History size approaching limit | Continue-as-New to reset history | | `EVENTS_HISTORY_LENGTH` | History event count approaching limit | Continue-as-New to reset history | -:::tip - -Handle different reasons differently. -For version upgrades, use the `AUTO_UPGRADE` versioning behavior option. -For history size reasons, you may want to continue without upgrading to stay on the same version. - -::: ### Limitations {#upgrade-on-can-limitations} From 6429e783995bfc9683c156ba3f17d5f82f2be448 Mon Sep 17 00:00:00 2001 From: Brandon Chavis Date: Fri, 23 Jan 2026 16:45:54 -0500 Subject: [PATCH 3/4] Add Worker tuning quick reference guide - Add new `worker-tuning-reference.mdx` with SDK defaults organized by resource type (compute, memory, IO) and metrics reference - Add to sidebar navigation after worker-performance - Add cross-references from worker-performance.mdx and workers.mdx This provides a quick reference companion to the comprehensive worker-performance guide, making it easy to look up SDK-specific defaults and relevant metrics by resource category. Co-Authored-By: Claude Opus 4.5 --- docs/develop/worker-performance.mdx | 1 + docs/develop/worker-tuning-reference.mdx | 215 +++++++++++++++++++++++ docs/encyclopedia/workers/workers.mdx | 1 + sidebars.js | 1 + 4 files changed, 218 insertions(+) create mode 100644 docs/develop/worker-tuning-reference.mdx diff --git a/docs/develop/worker-performance.mdx b/docs/develop/worker-performance.mdx index 4458937fe0..94efe80ae4 100644 --- a/docs/develop/worker-performance.mdx +++ b/docs/develop/worker-performance.mdx @@ -846,5 +846,6 @@ If, after adjusting the poller and executors count as specified earlier, you sti ## Related reading +- [Worker tuning quick reference](/develop/worker-tuning-reference) - SDK defaults and metrics by resource type - [Workers in production operation guide](https://temporal.io/blog/workers-in-production) - [Full set of SDK Metrics reference](/references/sdk-metrics) diff --git a/docs/develop/worker-tuning-reference.mdx b/docs/develop/worker-tuning-reference.mdx new file mode 100644 index 0000000000..130bd2071c --- /dev/null +++ b/docs/develop/worker-tuning-reference.mdx @@ -0,0 +1,215 @@ +--- +id: worker-tuning-reference +title: Worker tuning quick reference +sidebar_label: Worker tuning reference +description: A quick reference guide for Temporal Worker configuration defaults across SDKs, organized by resource type (compute, memory, IO) with key metrics for each. +toc_max_heading_level: 4 +keywords: + - worker tuning + - worker configuration + - sdk defaults + - worker metrics + - performance +tags: + - Workers + - Performance + - Reference +--- + +This page provides a quick reference for Worker configuration options and their default values across Temporal SDKs. +Use this guide alongside the comprehensive [Worker performance](/develop/worker-performance) documentation for detailed tuning guidance. + +Worker performance is constrained by three primary resources: + +| Resource | Description | +|----------|-------------| +| **Compute** | CPU-bound operations, concurrent Task execution | +| **Memory** | Workflow cache, thread pools | +| **IO** | Network calls to Temporal Service, polling | + +## How a Worker works + +Workers poll a [Task Queue](/workers#task-queues) in Temporal Cloud or a self-hosted Temporal Service, execute Tasks, and respond with the result. + +``` +┌─────────────────┐ Poll for Tasks ┌─────────────────┐ +│ Worker │ ◄─────────────────────── │ Temporal Service│ +│ - Workflows │ │ │ +│ - Activities │ ─────────────────────► │ │ +└─────────────────┘ Respond with results └─────────────────┘ +``` + +Multiple Workers can poll the same Task Queue, providing horizontal scalability. + +### How Worker failure recovery works + +When a Worker crashes or experiences a host outage: + +1. The Workflow Task times out +2. Another available Worker picks up the Task +3. The new Worker replays the Event History to reconstruct state +4. Execution continues from where it left off + +For more details on Worker architecture, see [What is a Temporal Worker?](/workers) + +## Compute settings + +Compute settings control how many Tasks a Worker can execute concurrently. + +### Compute configuration options + +| Setting | Description | +|---------|-------------| +| `MaxConcurrentWorkflowTaskExecutionSize` | Maximum concurrent Workflow Tasks | +| `MaxConcurrentActivityTaskExecutionSize` | Maximum concurrent Activity Tasks | +| `MaxConcurrentLocalActivityTaskExecutionSize` | Maximum concurrent Local Activities | +| `MaxWorkflowThreadCount` / `workflowThreadPoolSize` | Thread pool for Workflow execution | + +### Compute defaults by SDK + +| SDK | MaxConcurrentWorkflowTaskExecutionSize | MaxConcurrentActivityTaskExecutionSize | MaxConcurrentLocalActivityTaskExecutionSize | MaxWorkflowThreadCount | +|-----|----------------------------------------|----------------------------------------|---------------------------------------------|------------------------| +| **Go** | 1,000 | 1,000 | 1,000 | - | +| **Java** | 200 | 200 | 200 | 600 | +| **TypeScript** | 40 | 100 | 100 | 1 (reuseV8Context) | +| **Python** | 100 | 100 | 100 | - | +| **.NET** | 100 | 100 | 100 | - | + +### Resource-based slot suppliers + +Instead of fixed slot counts, you can use resource-based slot suppliers that automatically adjust available Task slots based on CPU and memory utilization. +For implementation details, see [Slot suppliers](/develop/worker-performance#slot-suppliers). + +## Memory settings + +Memory settings control the Workflow cache size and thread pool allocation. + +### Memory configuration options + +| Setting | Description | +|---------|-------------| +| `MaxCachedWorkflows` / `StickyWorkflowCacheSize` | Number of Workflows to keep in cache | +| `MaxWorkflowThreadCount` | Thread pool size | +| `reuseV8Context` (TypeScript) | Reuse V8 context for Workflows | + +### Memory defaults by SDK + +| SDK | MaxCachedWorkflows / StickyWorkflowCacheSize | +|-----|----------------------------------------------| +| **Go** | 10,000 | +| **Java** | 600 | +| **TypeScript** | Dynamic (e.g., 2000 for 4 GiB RAM) | +| **Python** | 1,000 | +| **.NET** | 10,000 | + +For cache tuning guidance, see [Workflow cache tuning](/develop/worker-performance#workflow-cache-tuning). + +## IO settings + +IO settings control the number of pollers and rate limits for Task Queue interactions. + +### IO configuration options + +| Setting | Description | +|---------|-------------| +| `MaxConcurrentWorkflowTaskPollers` | Number of concurrent Workflow pollers | +| `MaxConcurrentActivityTaskPollers` | Number of concurrent Activity pollers | +| `Namespace APS` | Actions per second limit for Namespace | +| `TaskQueueActivitiesPerSecond` | Activity rate limit per Task Queue | + +### IO defaults by SDK + +| SDK | MaxConcurrentWorkflowTaskPollers | MaxConcurrentActivityTaskPollers | Namespace APS | TaskQueueActivitiesPerSecond | +|-----|----------------------------------|----------------------------------|---------------|------------------------------| +| **Go** | 2 | 2 | 400 | Unlimited | +| **Java** | 5 | 5 | - | - | +| **TypeScript** | 10 | 10 | - | - | +| **Python** | 5 | 5 | - | - | +| **.NET** | 5 | 5 | - | - | + +### Poller autoscaling + +Use poller autoscaling to automatically adjust the number of concurrent polls based on workload. +For configuration details, see [Configuring poller options](/develop/worker-performance#configuring-poller-options). + +## Metrics reference by resource type + +Use these metrics to identify bottlenecks and guide tuning decisions. +For the complete metrics reference, see [SDK metrics](/references/sdk-metrics). + +### Compute-related metrics + +| Worker configuration option | SDK metric | +|-----------------------------|------------| +| `MaxConcurrentWorkflowTaskExecutionSize` | `worker_task_slots_available {worker_type = WorkflowWorker}` | +| `MaxConcurrentActivityTaskExecutionSize` | `worker_task_slots_available {worker_type = ActivityWorker}` | +| `MaxWorkflowThreadCount` | `workflow_active_thread_count` (Java only) | +| CPU-intensive logic | `workflow_task_execution_latency` | + +Also monitor your machine's CPU consumption (for example, `container_cpu_usage_seconds_total` in Kubernetes). + +### Memory-related metrics + +| Worker configuration option | SDK metric | +|-----------------------------|------------| +| `StickyWorkflowCacheSize` | `sticky_cache_total_forced_eviction`, `sticky_cache_size`, `sticky_cache_hit`, `sticky_cache_miss` | + +Also monitor your machine's memory consumption (for example, `container_memory_usage_bytes` in Kubernetes). + +### IO-related metrics + +| Worker configuration option | SDK metric | +|-----------------------------|------------| +| `MaxConcurrentWorkflowTaskPollers` | `num_pollers {poller_type = workflow_task}` | +| `MaxConcurrentActivityTaskPollers` | `num_pollers {poller_type = activity_task}` | +| Network latency | `request_latency {namespace, operation}` | + +### Task Queue metrics + +| Metric | Description | +|--------|-------------| +| `poll_success_sync_count` | Sync match rate (Tasks immediately assigned to Workers) | +| `approximate_backlog_count` | Approximate number of Tasks in a Task Queue | + +Task Queue statistics are also available via the `DescribeTaskQueue` API: +- `ApproximateBacklogCount` +- `ApproximateBacklogAge` +- `TasksAddRate` +- `TasksDispatchRate` +- `BacklogIncreaseRate` + +For more on Task Queue metrics, see [Available Task Queue information](/develop/worker-performance#task-queue-metrics). + +### Failure metrics + +| Metric | Description | +|--------|-------------| +| `long_request_failure` | Failures for long-running operations (polling, history retrieval) | +| `request_failure` | Failures for standard operations (Task completion responses) | + +Common failure codes: +- `RESOURCE_EXHAUSTED` - Rate limits exceeded +- `DEADLINE_EXCEEDED` - Operation timeout +- `NOT_FOUND` - Resource not found + +## Worker tuning tips + +1. **Scale test before production**: Validate your configuration under realistic load. +2. **Infrastructure matters**: Workers don't operate in a vacuum. Consider network latency, database performance, and external service dependencies. +3. **Tune and observe**: Make incremental changes and monitor metrics before making additional adjustments. +4. **Identify the bottleneck**: Use the [theory of constraints](https://en.wikipedia.org/wiki/Theory_of_constraints). Improving non-bottleneck resources won't improve overall throughput. + +For detailed tuning guidance, see: +- [Worker performance](/develop/worker-performance) +- [Worker deployment and performance best practices](/best-practices/worker) +- [Performance bottlenecks troubleshooting](/troubleshooting/performance-bottlenecks) + +## Related resources + +- [What is a Temporal Worker?](/workers) - Conceptual overview +- [Worker performance](/develop/worker-performance) - Comprehensive tuning guide +- [Worker deployment and performance](/best-practices/worker) - Best practices +- [SDK metrics reference](/references/sdk-metrics) - Complete metrics documentation +- [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) - Safe deployments +- [Workers in production](https://temporal.io/blog/workers-in-production) - Blog post +- [Introduction to Worker Tuning](https://temporal.io/blog/an-introduction-to-worker-tuning) - Blog post diff --git a/docs/encyclopedia/workers/workers.mdx b/docs/encyclopedia/workers/workers.mdx index 72d4529bd8..bd09700215 100644 --- a/docs/encyclopedia/workers/workers.mdx +++ b/docs/encyclopedia/workers/workers.mdx @@ -65,6 +65,7 @@ Therefore, a single Worker can handle millions of open Workflow Executions, assu **Operation guides:** - [How to tune Workers](/develop/worker-performance) +- [Worker tuning quick reference](/develop/worker-tuning-reference) - SDK defaults and metrics ## What is a Worker Identity? {#worker-identity} diff --git a/sidebars.js b/sidebars.js index 8e3fe05780..0b8c2217f5 100644 --- a/sidebars.js +++ b/sidebars.js @@ -311,6 +311,7 @@ module.exports = { 'develop/environment-configuration', 'develop/activity-retry-simulator', 'develop/worker-performance', + 'develop/worker-tuning-reference', 'develop/safe-deployments', 'develop/plugins-guide', ], From dbd750afe17c93cf913140e96b01327c73577758 Mon Sep 17 00:00:00 2001 From: flippedcoder Date: Wed, 28 Jan 2026 12:06:43 -0600 Subject: [PATCH 4/4] chore(worker versioning): removed CaN snippets --- .../worker-deployments/worker-versioning.mdx | 525 +++++++----------- 1 file changed, 191 insertions(+), 334 deletions(-) diff --git a/docs/production-deployment/worker-deployments/worker-versioning.mdx b/docs/production-deployment/worker-deployments/worker-versioning.mdx index 744889d3e5..76f4d24bc4 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning.mdx @@ -2,7 +2,9 @@ id: worker-versioning title: Worker Versioning sidebar_label: Worker Versioning -description: Use Worker Versioning to pin Workflow revisions to individual Worker Deployment Versions, avoiding the need for patching to support multiple code paths. +description: + Use Worker Versioning to pin Workflow revisions to individual Worker Deployment Versions, avoiding the need for + patching to support multiple code paths. slug: /production-deployment/worker-deployments/worker-versioning toc_max_heading_level: 4 keywords: @@ -18,8 +20,9 @@ tags: import { LANGUAGE_TAB_GROUP, getLanguageLabel } from '@site/src/constants/languageTabs'; import SdkTabs from '@site/src/components'; -Worker Versioning is a Temporal feature that allows you to confidently deploy new changes to the Workflows running on your Workers without breaking them. -Temporal enables this by helping you manage different builds or versions, formally called [Worker Deployment Versions](/worker-versioning#deployment-versions). +Worker Versioning is a Temporal feature that allows you to confidently deploy new changes to the Workflows running on +your Workers without breaking them. Temporal enables this by helping you manage different builds or versions, formally +called [Worker Deployment Versions](/worker-versioning#deployment-versions). Worker Versioning unlocks important benefits for users of [blue-green or rainbow deployments](#deployment-systems). @@ -28,26 +31,33 @@ Worker Versioning unlocks important benefits for users of [blue-green or rainbow - Instant rollback when you detect that a new Deployment Version is broken. - Improved error rates when adopting it. -In addition, Worker Versioning introduces **Workflow Pinning**. -For pinned Workflow Types, each execution runs entirely on the Worker Deployment Version where it started. -You need not worry about making breaking code changes to running, pinned Workflows. +In addition, Worker Versioning introduces **Workflow Pinning**. For pinned Workflow Types, each execution runs entirely +on the Worker Deployment Version where it started. You need not worry about making breaking code changes to running, +pinned Workflows. To use Workflow Pinning, we recommend using [rainbow deployments](#deployment-systems). :::tip + Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo.
- +
+ ::: :::note + Worker Versioning is currently available in Public Preview. Minimum versions: @@ -65,26 +75,39 @@ Self-hosted users: - Minimum Temporal CLI version [v1.4.1](https://github.com/temporalio/cli/releases/tag/v1.4.1) - Minimum Temporal Server version: [v1.29.1](https://github.com/temporalio/temporal/releases/tag/v1.29.1) - Minimum Temporal UI Version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0) - ::: + +::: ## Getting Started with Worker Versioning {#definition} To get started with Worker Versioning, you should understand some concepts around versioning and deployments. -- A **Worker Deployment** is a deployment or service across multiple versions. In a rainbow deployment, more than two active Deployment Versions can run at once. -- A **Worker Deployment Version** is a version of a deployment or service. It can have multiple Workers polling on multiple Task Queues, but they all run the same build. +- A **Worker Deployment** is a deployment or service across multiple versions. In a rainbow deployment, more than two + active Deployment Versions can run at once. +- A **Worker Deployment Version** is a version of a deployment or service. It can have multiple Workers polling on + multiple Task Queues, but they all run the same build. - A **Build ID**, in combination with a Worker Deployment name, identifies a single Worker Deployment Version. -- When a versioned worker polls on a task queue, that task queue becomes part of that Worker's version. That version's Worker Deployment controls how the task queue matches Workflow Tasks with Workers. -- Using **Workflow Pinning**, you can declare each Workflow type to have a **Versioning Behavior**, either Pinned or Auto-Upgrade. +- When a versioned worker polls on a task queue, that task queue becomes part of that Worker's version. That version's + Worker Deployment controls how the task queue matches Workflow Tasks with Workers. +- Using **Workflow Pinning**, you can declare each Workflow type to have a **Versioning Behavior**, either Pinned or + Auto-Upgrade. - A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version. - - An **Auto-Upgrade** Workflow will automatically move to a new code version as you roll it out, specifically its Target Worker Deployment Version (defined below). Therefore, Auto-Upgrade Workflows are not restricted to a single Deployment Version and need to be kept replay-safe manually, i.e. with [patching](/workflow-definition#workflow-versioning). - - Both Pinned and Auto-Upgrade Workflows are guaranteed to start only on the Current or Ramping Version of their Worker Deployment. + - An **Auto-Upgrade** Workflow will automatically move to a new code version as you roll it out, specifically its + Target Worker Deployment Version (defined below). Therefore, Auto-Upgrade Workflows are not restricted to a single + Deployment Version and need to be kept replay-safe manually, i.e. with + [patching](/workflow-definition#workflow-versioning). + - Both Pinned and Auto-Upgrade Workflows are guaranteed to start only on the Current or Ramping Version of their + Worker Deployment. - Pinned Workflows are designed for use with rainbow deployments. See [Deployment Systems](#deployment-systems). - Pinned Workflows don't need to be patched, as they run on the same worker and build until they complete. - - If you expect your Workflow to run longer than you want your Worker Deployment Versions to exist, you should mark your Workflow Type as Auto-Upgrade. -- Each Worker Deployment has a single [**Current Version**](/worker-versioning#versioning-definitions) which is where Workflows are routed to unless they were previously pinned on a different version. -- Each Worker Deployment can have a [**Ramping Version**](/worker-versioning#versioning-definitions) which is where a configurable percentage of Workflows are routed to unless they were previously pinned on a different version. -- For a given Workflow, its [**Target Worker Deployment Version**](/worker-versioning#versioning-definitions) is the version it will move to next. + - If you expect your Workflow to run longer than you want your Worker Deployment Versions to exist, you should mark + your Workflow Type as Auto-Upgrade. +- Each Worker Deployment has a single [**Current Version**](/worker-versioning#versioning-definitions) which is where + Workflows are routed to unless they were previously pinned on a different version. +- Each Worker Deployment can have a [**Ramping Version**](/worker-versioning#versioning-definitions) which is where a + configurable percentage of Workflows are routed to unless they were previously pinned on a different version. +- For a given Workflow, its [**Target Worker Deployment Version**](/worker-versioning#versioning-definitions) is the + version it will move to next. ## Setting up your deployment system {#deployment-systems} @@ -92,22 +115,34 @@ If you haven't already, you'll want to pick a container deployment solution for You also need to pick among three common deployment strategies: -- A **rolling deployment** strategy upgrades Workers in place with little control over how quickly they cut over and only a slow ability to roll Workers back. Rolling deploys have minimal footprint but tend to provide lower availability than the other strategies and are incompatible with Worker Versioning. -- A **blue-green deployment** strategy maintains two "colors," or Worker Deployment Versions simultaneously and can control how traffic is routed between them. This allows you to maximize your uptime with features like instant rollback and ramping. Worker Versioning enables the routing control that blue-green deployments need. -- A **rainbow deployment** strategy is like blue-green but with more colors, allowing Workflow Pinning. You can deploy new revisions of your Workflows freely while older versions drain. Using Worker Versioning, Temporal lets you know when all the Workflows of a given version are drained so that you can sunset it. +- A **rolling deployment** strategy upgrades Workers in place with little control over how quickly they cut over and + only a slow ability to roll Workers back. Rolling deploys have minimal footprint but tend to provide lower + availability than the other strategies and are incompatible with Worker Versioning. +- A **blue-green deployment** strategy maintains two "colors," or Worker Deployment Versions simultaneously and can + control how traffic is routed between them. This allows you to maximize your uptime with features like instant + rollback and ramping. Worker Versioning enables the routing control that blue-green deployments need. +- A **rainbow deployment** strategy is like blue-green but with more colors, allowing Workflow Pinning. You can deploy + new revisions of your Workflows freely while older versions drain. Using Worker Versioning, Temporal lets you know + when all the Workflows of a given version are drained so that you can sunset it. :::note -You also have the option to use the [Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to automatically enable rainbow deployments of your Workers if you're using Kubernetes. + +You also have the option to use the +[Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to automatically enable +rainbow deployments of your Workers if you're using Kubernetes. + ::: ## Configuring a Worker for Versioning -You'll need to add a few additional configuration parameters to your Workers to toggle on Worker Versioning. -There are three new parameters, with different names depending on the language: +You'll need to add a few additional configuration parameters to your Workers to toggle on Worker Versioning. There are +three new parameters, with different names depending on the language: - `UseVersioning`: This enables the Versioning functionality for this Worker. -- A `Version` to identify the revision that this Worker will be allowed to execute. This is a combination of a deployment name and a build ID number. -- (Optional) The [Default Versioning Behavior](#definition). If unset, you'll be required to specify the behavior on each Workflow. Or you can default to Pinned or Auto-Upgrade. +- A `Version` to identify the revision that this Worker will be allowed to execute. This is a combination of a + deployment name and a build ID number. +- (Optional) The [Default Versioning Behavior](#definition). If unset, you'll be required to specify the behavior on + each Workflow. Or you can default to Pinned or Auto-Upgrade. Follow the example for your SDK below: @@ -134,16 +169,11 @@ import io.temporal.common.VersioningBehavior; import io.temporal.common.WorkerDeploymentVersion; import io.temporal.worker.WorkerDeploymentOptions; - WorkerOptions.newBuilder() - .setDeploymentOptions( - WorkerDeploymentOptions.newBuilder() - .setVersion(new WorkerDeploymentVersion("llm_srv", "1.0")) - .setUseVersioning(true) - .setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) - .build()) - .build(); +WorkerOptions.newBuilder() .setDeploymentOptions( WorkerDeploymentOptions.newBuilder() .setVersion(new +WorkerDeploymentVersion("llm_srv", "1.0")) .setUseVersioning(true) +.setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) .build()) .build(); -``` +```` ```python @@ -163,7 +193,7 @@ Worker( default_versioning_behavior=VersioningBehavior.UNSPECIFIED ), ) -``` +```` @@ -216,64 +246,72 @@ worker = Temporalio::Worker.new( - - ## Choosing a Versioning Behavior {#choosing-behavior} The right versioning behavior depends on how long your Workflows run relative to your deployment frequency. ### Decision guide {#decision-guide} -| Workflow Duration | Uses Continue-as-New? | Recommended Behavior | Patching Required? | -|-------------------|----------------------|---------------------|-------------------| -| **Short** (completes before next deploy) | N/A | `PINNED` | Never | -| **Medium** (spans multiple deploys) | No | `AUTO_UPGRADE` | Yes | -| **Long** (weeks to years) | Yes | `PINNED` + [upgrade on CaN](#upgrade-on-continue-as-new) | Never | -| **Long** (weeks to years) | No | `AUTO_UPGRADE` + patching | Yes | +| Workflow Duration | Uses Continue-as-New? | Recommended Behavior | Patching Required? | +| ---------------------------------------- | --------------------- | -------------------------------------------------------- | ------------------ | +| **Short** (completes before next deploy) | N/A | `PINNED` | Never | +| **Medium** (spans multiple deploys) | No | `AUTO_UPGRADE` | Yes | +| **Long** (weeks to years) | Yes | `PINNED` + [upgrade on CaN](#upgrade-on-continue-as-new) | Never | +| **Long** (weeks to years) | No | `AUTO_UPGRADE` + patching | Yes | ### Examples by Workflow type {#behavior-examples} -| Workflow Type | Duration | Recommended Behavior | Notes | -|---------------|----------|---------------------|-------| -| Order processing | Minutes | `PINNED` | Completes before next deploy | -| Payment retry | Hours | `PINNED` or `AUTO_UPGRADE` | Depends on deploy frequency | -| Subscription billing | Days | `AUTO_UPGRADE` | May span multiple deploys | -| Customer entity | Months-Years | `PINNED` + upgrade on CaN | Uses Continue-as-New pattern | -| AI agent / Chatbot | Weeks | `PINNED` + upgrade on CaN | Long sleeps, uses CaN | -| Compliance audit | Months | `AUTO_UPGRADE` + patching | Cannot use CaN (needs full history) | +| Workflow Type | Duration | Recommended Behavior | Notes | +| -------------------- | ------------ | -------------------------- | ----------------------------------- | +| Order processing | Minutes | `PINNED` | Completes before next deploy | +| Payment retry | Hours | `PINNED` or `AUTO_UPGRADE` | Depends on deploy frequency | +| Subscription billing | Days | `AUTO_UPGRADE` | May span multiple deploys | +| Customer entity | Months-Years | `PINNED` + upgrade on CaN | Uses Continue-as-New pattern | +| AI agent / Chatbot | Weeks | `PINNED` + upgrade on CaN | Long sleeps, uses CaN | +| Compliance audit | Months | `AUTO_UPGRADE` + patching | Cannot use CaN (needs full history) | :::info Long-running Workflows with Continue-as-New -If your Workflow uses Continue-as-New to manage history size, you can upgrade to new Worker Deployment Versions at the CaN boundary without patching. -See [Upgrading on Continue-as-New](#upgrade-on-continue-as-new) below. +If your Workflow uses Continue-as-New to manage history size, you can upgrade to new Worker Deployment Versions at the +CaN boundary without patching. See [Upgrading on Continue-as-New](#upgrade-on-continue-as-new) below. ::: ### Default Versioning Behavior Considerations -If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning. Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`. +If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning. +Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`. -In general, each Workflow Type should be annotated as Auto-Upgrade or Pinned. If all of your Workflows will be short-running for the foreseeable future, you can default to Pinned. +In general, each Workflow Type should be annotated as Auto-Upgrade or Pinned. If all of your Workflows will be +short-running for the foreseeable future, you can default to Pinned. -Many users who are migrating to Worker Versioning will start by defaulting to Auto-Upgrade until they have had time to annotate their Workflows. -This default is the most similar to the legacy behavior. -Once each Workflow Type is annotated, you can remove the `DefaultVersioningBehavior`. +Many users who are migrating to Worker Versioning will start by defaulting to Auto-Upgrade until they have had time to +annotate their Workflows. This default is the most similar to the legacy behavior. Once each Workflow Type is annotated, +you can remove the `DefaultVersioningBehavior`. -There is a possibility of a queue blocking limitation for new or Auto-Upgrade Workflows if there is a ramp, but one of the Current or Ramping versions is down or doesn't have enough capacity. This leads to other versions not getting Tasks or slowing down. +There is a possibility of a queue blocking limitation for new or Auto-Upgrade Workflows if there is a ramp, but one of +the Current or Ramping versions is down or doesn't have enough capacity. This leads to other versions not getting Tasks +or slowing down. -For example, you have a Current Version and a Ramping Version at 50%. If all of your Current Version Workers go down, you would expect at least 50% of new Workflows to go to the Ramping Version. This won't happen because the Tasks for the Current Version are blocking the queue. +For example, you have a Current Version and a Ramping Version at 50%. If all of your Current Version Workers go down, +you would expect at least 50% of new Workflows to go to the Ramping Version. This won't happen because the Tasks for the +Current Version are blocking the queue. :::note -Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow default to Auto-Upgrade behavior and not Unspecified. -::: -You also want to make sure you understand how your Activities are going to work across different Worker Deployment Versions. Refer to the [Worker Versioning Activitiy behavior docs](/worker-versioning#actvity-behavior-across-versions) for more details. +Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow default to Auto-Upgrade behavior and not +Unspecified. + +::: +You also want to make sure you understand how your Activities are going to work across different Worker Deployment +Versions. Refer to the [Worker Versioning Activitiy behavior docs](/worker-versioning#actvity-behavior-across-versions) +for more details. ## Rolling out changes with the CLI -Next, deploy your Worker with the additional configuration parameters. -Before making any Workflow revisions, you can use the `temporal` CLI to check which of your Worker versions are currently polling: +Next, deploy your Worker with the additional configuration parameters. Before making any Workflow revisions, you can use +the `temporal` CLI to check which of your Worker versions are currently polling: You can view the Versions that are part of a Deployment with `temporal worker deployment describe`: @@ -281,7 +319,8 @@ You can view the Versions that are part of a Deployment with `temporal worker de temporal worker deployment describe --name="$MY_DEPLOYMENT" ``` -To activate a Deployment Version, use `temporal worker deployment set-current-version`, specifying the deployment name and a Build ID: +To activate a Deployment Version, use `temporal worker deployment set-current-version`, specifying the deployment name +and a Build ID: ```bash temporal worker deployment set-current-version \ @@ -289,7 +328,8 @@ temporal worker deployment set-current-version \ --build-id "YourBuildID" ``` -To ramp a Deployment Version up to some percentage of your overall Worker fleet, use `set-ramping version`, with the same parameters and a ramping percentage: +To ramp a Deployment Version up to some percentage of your overall Worker fleet, use `set-ramping version`, with the +same parameters and a ramping percentage: ```bash temporal worker deployment set-ramping-version \ @@ -316,8 +356,8 @@ Versioning Info: ## Marking a Workflow Type as Pinned -You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. -This will cause it to remain on its original deployed version: +You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. This will cause it +to remain on its original deployed version: @@ -337,14 +377,10 @@ public interface HelloWorld { String hello(); } -public static class HelloWorldImpl implements HelloWorld { - @Override - @WorkflowVersioningBehavior(VersioningBehavior.PINNED) - public String hello() { - return "Hello, World!"; - } -} -``` +public static class HelloWorldImpl implements HelloWorld { @Override +@WorkflowVersioningBehavior(VersioningBehavior.PINNED) public String hello() { return "Hello, World!"; } } + +```` ```python @@ -353,7 +389,7 @@ class HelloWorld: @workflow.run async def run(self): return "hello world!" -``` +```` @@ -398,7 +434,8 @@ end ## Moving a pinned Workflow -Sometimes you'll need to manually move a set of pinned Workflows off of a version that has a bug to a version with the fix. +Sometimes you'll need to manually move a set of pinned Workflows off of a version that has a bug to a version with the +fix. If you need to move a pinned Workflow to a new version, use `temporal workflow update-options`: @@ -420,14 +457,14 @@ temporal workflow update-options \ --versioning-override-build-id "$FIXED_BUILD_ID" ``` -In this scenario, you may also need to use the other [Versioning APIs](/workflow-definition#workflow-versioning) to patch -your Workflow in the "fixed" build, so that your target Worker can handle the moved Workflows correctly. -If you made a [version-incompatible change](/workflow-definition#deterministic-constraints) to your Workflow, and you -want to roll back to an earlier version, it's not possible to patch it. Considering using [Workflow Reset](/workflow-execution/event#reset) -along with your move. +In this scenario, you may also need to use the other [Versioning APIs](/workflow-definition#workflow-versioning) to +patch your Workflow in the "fixed" build, so that your target Worker can handle the moved Workflows correctly. If you +made a [version-incompatible change](/workflow-definition#deterministic-constraints) to your Workflow, and you want to +roll back to an earlier version, it's not possible to patch it. Considering using +[Workflow Reset](/workflow-execution/event#reset) along with your move. -"Reset-with-Move" allows you to atomically Reset your Workflow and set a Versioning Override on the newly reset Workflow, -so when it resumes execution, all new Workflow Tasks will be executed on your new Worker. +"Reset-with-Move" allows you to atomically Reset your Workflow and set a Versioning Override on the newly reset +Workflow, so when it resumes execution, all new Workflow Tasks will be executed on your new Worker. ```bash temporal workflow reset with-workflow-update-options \ @@ -441,11 +478,14 @@ temporal workflow reset with-workflow-update-options \ ## Migrating a Workflow from Pinned to Auto-Upgrade -There may be times when you need to migrate your Workflow from Pinned to Auto-Upgrade because you configured your Workflow Type with the wrong behavior or you've pinned a really long-running Workflow by mistake. +There may be times when you need to migrate your Workflow from Pinned to Auto-Upgrade because you configured your +Workflow Type with the wrong behavior or you've pinned a really long-running Workflow by mistake. -Pinned Workflows can block version drainage, especially when they run for a long time. You could move the Workflow to a new build, but that would just push the problem to the next build. +Pinned Workflows can block version drainage, especially when they run for a long time. You could move the Workflow to a +new build, but that would just push the problem to the next build. -In order to make this change, you need to change the versioning behavior for your Workflow from Pinned to Auto-Upgrade. You can use `temporal workflow update-options` for this: +In order to make this change, you need to change the versioning behavior for your Workflow from Pinned to Auto-Upgrade. +You can use `temporal workflow update-options` for this: ```bash temporal workflow update-options \ @@ -470,14 +510,20 @@ temporal workflow update-options \ ``` :::note -When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the Workflow's Target Version is different from the earlier Pinned Version, you should make sure you [patch](/patching#patching) the Workflow code. + +When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the +Workflow's Target Version is different from the earlier Pinned Version, you should make sure you +[patch](/patching#patching) the Workflow code. + ::: ## Upgrading on Continue-as-New {#upgrade-on-continue-as-new} -Long-running Workflows that use [Continue-as-New](/workflow-execution/continue-as-new) can upgrade to newer Worker Deployment Versions at Continue-as-New boundaries without requiring patching. +Long-running Workflows that use [Continue-as-New](/workflow-execution/continue-as-new) can upgrade to newer Worker +Deployment Versions at Continue-as-New boundaries without requiring patching. This pattern is ideal for: + - **Entity Workflows** that run for months or years - **Batch processing** Workflows that checkpoint with Continue-as-New - **AI agent Workflows** with long sleeps waiting for user input @@ -490,262 +536,64 @@ This feature is in Public Preview as an experimental SDK-level option. ### How it works {#upgrade-on-can-how-it-works} -By default, Pinned Workflows stay on their original Worker Deployment Version even when they Continue-as-New. -With the upgrade option enabled: +By default, Pinned Workflows stay on their original Worker Deployment Version even when they Continue-as-New. With the +upgrade option enabled: 1. Each Workflow run remains pinned to its version (no patching needed during a run) 2. The Temporal Server suggests Continue-as-New when a new version becomes available -3. When the Workflow performs Continue-as-New with the upgrade option, the new run starts on the Current or Ramping version +3. When the Workflow performs Continue-as-New with the upgrade option, the new run starts on the Current or Ramping + version ### Checking for new versions {#checking-for-new-versions} -When a new Worker Deployment Version becomes Current or Ramping, active Workflows can detect this through `continue_as_new_suggested`. -Check for the `TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED` reason: - - - - -```go -import ( - "go.temporal.io/sdk/workflow" -) - -func MyEntityWorkflow(ctx workflow.Context, state EntityState) error { - for { - // Your workflow logic here - - // Check if we should continue as new - if workflow.IsContinueAsNewSuggested(ctx) { - info := workflow.GetInfo(ctx) - for _, reason := range info.ContinueAsNewSuggestedReasons { - if reason == workflow.ContinueAsNewSuggestedReasonTargetVersionChanged { - // A new Worker Deployment Version is available - // Continue-as-New with upgrade to the new version - return workflow.NewContinueAsNewError( - ctx, - MyEntityWorkflow, - state, - workflow.WithContinueAsNewVersioningBehavior(workflow.VersioningBehaviorAutoUpgrade), - ) - } - } - // Other CaN reasons (history size) - continue without upgrading - return workflow.NewContinueAsNewError(ctx, MyEntityWorkflow, state) - } - - // Wait for signals, timers, etc. - selector := workflow.NewSelector(ctx) - selector.AddReceive(workflow.GetSignalChannel(ctx, "update"), func(c workflow.ReceiveChannel, more bool) { - // Handle signal - }) - selector.Select(ctx) - } -} -``` - - - - -```python -from datetime import timedelta -from temporalio import workflow -from temporalio.workflow import ContinueAsNewSuggestedReason, VersioningBehavior - -@workflow.defn -class MyEntityWorkflow: - @workflow.run - async def run(self, state: EntityState) -> None: - while True: - # Your workflow logic here - - # Check if we should continue as new - if workflow.continue_as_new_suggested(): - info = workflow.info() - for reason in info.continue_as_new_suggested_reasons: - if reason == ContinueAsNewSuggestedReason.TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED: - # A new Worker Deployment Version is available - # Continue-as-New with upgrade to the new version - workflow.continue_as_new( - args=[state], - versioning_behavior=VersioningBehavior.AUTO_UPGRADE, - ) - - # Other CaN reasons (history size) - continue without upgrading - workflow.continue_as_new(args=[state]) - - # Wait for signals, timers, etc. - await workflow.wait_condition( - lambda: self.has_pending_work or workflow.continue_as_new_suggested(), - timeout=timedelta(hours=1), - ) -``` - - - - -```typescript -import { - continueAsNew, - continueAsNewSuggested, - workflowInfo, - condition, - ContinueAsNewSuggestedReason, -} from '@temporalio/workflow'; - -export async function myEntityWorkflow(state: EntityState): Promise { - while (true) { - // Your workflow logic here - - // Check if we should continue as new - if (continueAsNewSuggested()) { - const info = workflowInfo(); - for (const reason of info.continueAsNewSuggestedReasons ?? []) { - if (reason === ContinueAsNewSuggestedReason.TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED) { - // A new Worker Deployment Version is available - // Continue-as-New with upgrade to the new version - await continueAsNew(state, { - versioningBehavior: 'AUTO_UPGRADE', - }); - } - } - // Other CaN reasons (history size) - continue without upgrading - await continueAsNew(state); - } - - // Wait for signals, timers, etc. - await condition(() => hasPendingWork || continueAsNewSuggested(), '1h'); - } -} -``` - - - - -```java -import io.temporal.workflow.Workflow; -import io.temporal.workflow.ContinueAsNewOptions; -import io.temporal.api.enums.v1.ContinueAsNewSuggestedReason; -import io.temporal.api.enums.v1.VersioningBehavior; - -public class MyEntityWorkflowImpl implements MyEntityWorkflow { - @Override - public void run(EntityState state) { - while (true) { - // Your workflow logic here - - // Check if we should continue as new - if (Workflow.isContinueAsNewSuggested()) { - var info = Workflow.getInfo(); - for (var reason : info.getContinueAsNewSuggestedReasons()) { - if (reason == ContinueAsNewSuggestedReason.TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED) { - // A new Worker Deployment Version is available - Workflow.continueAsNew( - ContinueAsNewOptions.newBuilder() - .setVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) - .build(), - state - ); - } - } - // Other CaN reasons - continue without upgrading - Workflow.continueAsNew(state); - } - - // Wait for signals, timers, etc. - Workflow.await(() -> hasPendingWork || Workflow.isContinueAsNewSuggested()); - } - } -} -``` - - - - -```csharp -using Temporalio.Workflows; - -[Workflow] -public class MyEntityWorkflow -{ - [WorkflowRun] - public async Task RunAsync(EntityState state) - { - while (true) - { - // Your workflow logic here - - // Check if we should continue as new - if (Workflow.ContinueAsNewSuggested) - { - var info = Workflow.Info; - foreach (var reason in info.ContinueAsNewSuggestedReasons) - { - if (reason == ContinueAsNewSuggestedReason.TargetWorkerDeploymentVersionChanged) - { - // A new Worker Deployment Version is available - throw Workflow.CreateContinueAsNewException( - wf => wf.RunAsync(state), - new() { VersioningBehavior = VersioningBehavior.AutoUpgrade }); - } - } - // Other CaN reasons - continue without upgrading - throw Workflow.CreateContinueAsNewException( - wf => wf.RunAsync(state)); - } - - // Wait for signals, timers, etc. - await Workflow.WaitConditionAsync( - () => hasPendingWork || Workflow.ContinueAsNewSuggested, - TimeSpan.FromHours(1)); - } - } -} -``` - - - +When a new Worker Deployment Version becomes Current or Ramping, active Workflows can detect this through +`continue_as_new_suggested`. Check for the `TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED` reason: ### Continue-as-New suggested reasons {#can-suggested-reasons} The `continue_as_new_suggested` mechanism can return multiple reasons: -| Reason | Description | Action | -|--------|-------------|--------| +| Reason | Description | Action | +| ------------------------------------------ | -------------------------------------------- | ---------------------------------- | | `TARGET_WORKER_DEPLOYMENT_VERSION_CHANGED` | A new Worker Deployment Version is available | Use upgrade option when continuing | -| `EVENTS_HISTORY_SIZE` | History size approaching limit | Continue-as-New to reset history | -| `EVENTS_HISTORY_LENGTH` | History event count approaching limit | Continue-as-New to reset history | - +| `EVENTS_HISTORY_SIZE` | History size approaching limit | Continue-as-New to reset history | +| `EVENTS_HISTORY_LENGTH` | History event count approaching limit | Continue-as-New to reset history | ### Limitations {#upgrade-on-can-limitations} :::caution Current Limitations -- **Lazy moving only:** Workflows must wake up naturally to receive the Continue-as-New suggestion. - Sleeping Workflows won't be proactively signaled (planned for a future release). -- **Interface compatibility:** When continuing as new to a different version, ensure your Workflow input format is compatible. - If incompatible, the new run may fail on its first Workflow Task. +- **Lazy moving only:** Workflows must wake up naturally to receive the Continue-as-New suggestion. Sleeping Workflows + won't be proactively signaled (planned for a future release). +- **Interface compatibility:** When continuing as new to a different version, ensure your Workflow input format is + compatible. If incompatible, the new run may fail on its first Workflow Task. ::: - ## Sunsetting an old Deployment Version A Worker Deployment Version moves through the following states: -1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never becomes Active, it will never be Draining or Drained. -2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing Auto-Upgrade Workflows. -3. **Draining**: The version stopped being Current or Ramping, and it has open pinned Workflows running on it. It is possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated periodically. +1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never + becomes Active, it will never be Draining or Drained. +2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing Auto-Upgrade + Workflows. +3. **Draining**: The version stopped being Current or Ramping, and it has open pinned Workflows running on it. It is + possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated + periodically. 4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed. -You can see these statuses when you describe a Worker Deployment in the `WorkerDeploymentVersionStatus` of each `VersionSummary`, or by describing the version directly. -When a version is Draining or Drained, that is displayed in a value called `DrainageStatus`. -Periodically, the Temporal Service will refresh this status by counting any open pinned Workflows using that version. +You can see these statuses when you describe a Worker Deployment in the `WorkerDeploymentVersionStatus` of each +`VersionSummary`, or by describing the version directly. When a version is Draining or Drained, that is displayed in a +value called `DrainageStatus`. Periodically, the Temporal Service will refresh this status by counting any open pinned +Workflows using that version. -On each refresh, `DrainageInfo.last_checked_time` is updated. -Eventually, `DrainageInfo` will report that the version is fully drained. -At this point, no Workflows are still running on that version and no more will be automatically routed to it, so you can consider shutting down the running Workers. +On each refresh, `DrainageInfo.last_checked_time` is updated. Eventually, `DrainageInfo` will report that the version is +fully drained. At this point, no Workflows are still running on that version and no more will be automatically routed to +it, so you can consider shutting down the running Workers. -You can monitor this by checking `WorkerDeploymentInfo.VersionSummaries` or with `temporal worker deployment describe-version`: +You can monitor this by checking `WorkerDeploymentInfo.VersionSummaries` or with +`temporal worker deployment describe-version`: ```bash temporal worker deployment describe-version \ @@ -769,7 +617,8 @@ Task Queues: hello-world workflow ``` -If you have implemented [Queries](/sending-messages#sending-queries) on closed pinned Workflows, you may need to keep some Workers running to handle them. +If you have implemented [Queries](/sending-messages#sending-queries) on closed pinned Workflows, you may need to keep +some Workers running to handle them. ### Adding a pre-deployment test @@ -862,16 +711,24 @@ handle = env.client.start_workflow( ## Garbage collection -Worker Deployments are never garbage collected, but *Worker Deployment Versions* (often referred to as Versions, Worker Versions, Deployment Versions) are. +Worker Deployments are never garbage collected, but _Worker Deployment Versions_ (often referred to as Versions, Worker +Versions, Deployment Versions) are. -Versions are deleted to keep the total number of versions in one Worker Deployment less than or equal to [`matching.maxVersionsInDeployment`](https://github.com/temporalio/temporal/blob/a3a53266c002ae33b630a41977274f8b5b587031/common/dynamicconfig/constants.go#L1317-L1321), which is currently set to 100 in Temporal Cloud, but that's a conservative number and it could be increased if needed. +Versions are deleted to keep the total number of versions in one Worker Deployment less than or equal to +[`matching.maxVersionsInDeployment`](https://github.com/temporalio/temporal/blob/a3a53266c002ae33b630a41977274f8b5b587031/common/dynamicconfig/constants.go#L1317-L1321), +which is currently set to 100 in Temporal Cloud, but that's a conservative number and it could be increased if needed. -For example, when you deploy your 101st Worker version in a Worker Deployment, the server looks at the oldest drained version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it. If that version still has pollers, the server will try the next oldest version. -If none of the 100 versions are eligible for deletion (ie. none of them are drained with no pollers), then no version will be deleted and the poll from the 101st version would fail. +For example, when you deploy your 101st Worker version in a Worker Deployment, the server looks at the oldest drained +version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it. If that version +still has pollers, the server will try the next oldest version. If none of the 100 versions are eligible for deletion +(ie. none of them are drained with no pollers), then no version will be deleted and the poll from the 101st version +would fail. -At that point, to successfully deploy your 101st version, you would need to increase `matching.maxVersionsInDeployment` or stop polling from one of the old drained versions to make it eligible for clean up. +At that point, to successfully deploy your 101st version, you would need to increase `matching.maxVersionsInDeployment` +or stop polling from one of the old drained versions to make it eligible for clean up. -If you want to re-deploy a previously deleted version, start polling with a Worker that has the same build ID and Deployment Name as the deleted version and the server will recreate it. +If you want to re-deploy a previously deleted version, start polling with a Worker that has the same build ID and +Deployment Name as the deleted version and the server will recreate it. -This covers the complete lifecycle of working with Worker Versioning. -We are continuing to improve this feature, and we welcome any feedback or feature requests using the sidebar link! +This covers the complete lifecycle of working with Worker Versioning. We are continuing to improve this feature, and we +welcome any feedback or feature requests using the sidebar link!