diff --git a/docs/content/en/docs/5_architecture_and_components/engine/design.md b/docs/content/en/docs/5_architecture_and_components/engine/design.md index ad2f7f7fe..8e520a3a4 100644 --- a/docs/content/en/docs/5_architecture_and_components/engine/design.md +++ b/docs/content/en/docs/5_architecture_and_components/engine/design.md @@ -52,12 +52,9 @@ This pattern ensures atomicity - either all changes succeed and are persisted, o ### Repository Abstraction -The engine doesn't directly interact with Git repositories. Instead, it: - -- Opens repositories through the **cache layer** -- Works with repository abstractions that hide storage implementation details -- Delegates all storage operations to repository adapters -- Maintains separation between business logic and storage mechanisms +The engine does not directly interact with Git repositories. Instead, it opens repositories through the **cache layer** and works with +repository abstractions that hide storage implementation details. It also delegates all storage operations to repository adapters. +Additionally, it maintains separation between business logic and storage mechanisms. ### Concurrency Model @@ -75,31 +72,21 @@ The engine uses **per-package mutexes** to prevent concurrent modifications to t **Optimistic Locking:** -For update operations, the engine uses **Kubernetes resource versions**: - -- Client must provide current resource version in update request -- Engine compares provided version with actual version -- Returns conflict error if versions don't match -- Forces client to re-read and retry with latest version -- Prevents lost updates when multiple clients modify same package +For update operations, the engine uses **Kubernetes resource versions**. The client must provide current resource version in update +request. Then the engine compares provided version with actual version and returns a conflict error if the versions do not match. This +forces the client to re-read and retry with the latest version. This way lost updates are prevented when multiple clients modify the same +package. **Draft Isolation:** -Drafts provide natural concurrency isolation: - -- Each draft is a separate workspace in the repository -- Multiple drafts can exist for different workspaces simultaneously -- Drafts don't interfere with each other until closed -- Closing a draft is an atomic operation +Drafts provide natural concurrency isolation. Each draft is a separate workspace in the repository. This means that multiple drafts can +exist for different workspaces simultaneously and drafts do not interfere with each other until closed. Closing a draft is an atomic +operation. **Read Operations:** -List and Get operations are **lock-free**: - -- Read from cached repository state -- No locking required for queries -- May see slightly stale data during cache refresh -- Eventually consistent with repository state +List and Get operations are **lock-free**, they read from cached repository state. No locking is required for queries. You may see +slightly stale data during cache refresh but eventually it will be consistent with the repository state. **Concurrency characteristics:** - **Single package**: Only one write operation at a time (mutex protected) @@ -128,44 +115,41 @@ The Engine exposes a single interface (`CaDEngine`) with operations grouped by r - **ObjectCache**: Access the watcher manager for real-time change notifications - **OpenRepository**: Internal helper to access repositories through the cache -**Interface characteristics:** +**Interface characteristics** -- All operations are **context-aware** for cancellation and tracing -- Operations accept **API objects** (porchapi types) and return **repository abstractions** -- The interface is **synchronous** - operations complete before returning -- Errors are returned directly rather than stored in status fields +All operations are **context-aware** for cancellation and tracing. Operations accept **API objects** (porchapi types) and return +**repository abstractions**. The interface is **synchronous**, which means that operations complete before returning. Additionally, errors +are returned directly rather than stored in status fields. ## Package Lifecycle State Machine The Engine enforces a strict state machine for package revision lifecycle: -![Package Lifecycle Workflow](/static/images/porch/flowchart.drawio.svg) +![Package Lifecycle Workflow](/static/images/porch/lifecycle-flowchart.drawio.svg) ### State Transition Rules -**Draft State:** -- Can be created directly or as default when no lifecycle specified -- Allows full modifications: tasks, resources, metadata, lifecycle -- Can transition to: Proposed, Published -- Cannot be created with Published or DeletionProposed lifecycle - -**Proposed State:** -- Indicates package revision is ready for review -- Allows modifications like Draft -- Can transition to: Draft (for rework), Published (for approval) -- Typically used in approval workflows - -**Published State:** -- **Immutable** - no resource or task modifications allowed -- Only metadata (labels, annotations) and lifecycle can be updated -- Can transition to: DeletionProposed -- Represents the deployed, production-ready state - -**DeletionProposed State:** -- Marks package revision for deletion -- Considered "published" for lifecycle checks -- Final state before actual deletion -- Used to signal intent to remove while maintaining audit trail +#### Draft State + +This state can be created directly, or as default when no lifecycle is specified. It allows full modifications of tasks, resources, +metadata or lifecycle. From draft state packages can transition to either Proposed or Published. However, they cannot be created with +Published or DeletionProposed lifecycle. + +#### Proposed State + +This state indicates that the package revision is ready for review. From the proposed state, packages can transition to either Draft +(for rework) or Published (for approval). This is typically used in approval workflows. + +#### Published State + +Packages in this state are immutable, meaning that no resource or task modifications are allowed to them. Only metadata (labels, +annotations) and lifecycle can be updated. From the published state, packages can transition to DeletionProposed. The published state +represents the deployed, production-ready state. + +#### DeletionProposed State + +This state marks the package revision for deletion. For lifecycle checks, packages in these state are considered "published". This is the +final state before actual deletion and it is used to signal intent to remove the package while maintaining audit trail. ### Lifecycle Enforcement diff --git a/docs/content/en/docs/5_architecture_and_components/engine/functionality/draft-commit-orchestration.md b/docs/content/en/docs/5_architecture_and_components/engine/functionality/draft-commit-orchestration.md index 96429c95a..a8971c1cb 100644 --- a/docs/content/en/docs/5_architecture_and_components/engine/functionality/draft-commit-orchestration.md +++ b/docs/content/en/docs/5_architecture_and_components/engine/functionality/draft-commit-orchestration.md @@ -271,16 +271,11 @@ Repository.UpdatePackageRevision Return Draft Handle ``` -**Draft from existing:** -- Loads current package revision content -- Creates mutable draft workspace -- Copies resources to draft for modification -- Preserves original revision (immutable) +In this case, the current package revision of the content is loaded and a mutable draft +workspace is created. Resources are copied to draft for modification, while the original revision is preserved (immutable) -**Draft operations:** -- Same as creation draft -- Can modify resources, lifecycle, metadata -- Changes isolated until closed +**Draft operations:** The same as creation draft, meaning resources, lifecycle and metadata can be modified. +The changes are isolated until closed. ### Mutation Application @@ -303,11 +298,8 @@ TaskHandler.DoPRMutations - **Lifecycle changes**: Handled separately (UpdateLifecycle) - **Metadata changes**: Handled after draft closure -**Mutation process:** -- Compares old and new package revision specs -- Identifies new tasks to apply -- Delegates to task handler for execution -- Returns modified draft +Old and new package revision specs are compared and new tasks to apply are identified. These tasks are delegated to the +task handler for execution after which a modified draft is returned. ### Metadata-Only Update Path @@ -325,16 +317,11 @@ UpdatePackageRevision (Published/DeletionProposed) Return Updated PR ``` -**Metadata-only updates:** -- For Published and DeletionProposed package revisions -- No draft-commit workflow needed -- Direct metadata update on immutable revision -- Only labels, annotations, finalizers, owner references +These are for Published and DeletionProposed package revisions, no draft-commit workflow is needed. This means direct metadata +update on immutable revision. Only labels, annotations, finalizers, owner references are updated. -**Why metadata-only:** -- Published packages are immutable (content cannot change) -- Metadata doesn't affect package content -- Allows operational updates without content changes +Published packages are immutable, meaning that the content cannot change. Metadata, however, does not affect package content. +This allows operational updates without content changes. ## Update Package Resources Workflow @@ -401,21 +388,17 @@ TaskHandler.DoPRResourceMutations Return RenderStatus ``` -**Resource mutation process:** -- Updates package resource content directly -- Executes render task (runs KRM functions) -- Returns render status with function results -- No lifecycle change (resources updated in-place) - -**Render execution:** -- Runs configured KRM function pipeline -- Functions can validate, transform, generate resources -- Results returned in RenderStatus -- **Render failure handling**: - - Default behavior: Render errors prevent draft closure (no resources persisted) - - With `porch.kpt.dev/push-on-render-failure: "true"` annotation: Draft is closed even on render failure - - The behavior of partially-rendered resources can be further controlled via Kptfile annotations (see [kpt documentation](https://kpt.dev/book/04-using-functions/#debugging-render-failures)) - - Error is always returned to caller regardless of persistence behavior +Package resource content is updated directly and the render task is executed by running KRM functions. Once done, +the render status with function results is returned. This does not involve a lifecycle change, as the resources are updated in-place. + +The render is executed by running a configured KRM function pipeline. These functions can validate, transform and generate resources. +The results are returned in RenderStatus. + +In case of render failure, the default behavior is for render errors to prevent draft closure (no resources persisted). +With the `porch.kpt.dev/push-on-render-failure: "true"` annotation, the draft is closed even on render failure. The behavior of +partially-rendered resources can be further controlled via Kptfile annotations +(see [kpt documentation](https://kpt.dev/book/04-using-functions/#debugging-render-failures)). The error is always returned to +caller regardless of persistence behavior. ### Persisting Resources on Render Failure @@ -439,12 +422,13 @@ The `porch.kpt.dev/push-on-render-failure` annotation enables saving work-in-pro kubectl annotate packagerevision porch.kpt.dev/push-on-render-failure=true ``` -**Important notes:** +{{% alert title="Note" color="primary" %}} - Only applies to Draft PackageRevisions during resource updates (via `UpdatePackageResources`) - Does not apply to package creation operations (init, clone, edit, copy) - Error is always returned even when resources are persisted - The behavior of partially-rendered resources can be further controlled via Kptfile annotations (see [kpt documentation](https://kpt.dev/book/04-using-functions/#debugging-render-failures)) -- In rare cases (e.g., internal errors during resource persistence), push may be prevented regardless of the annotation +- In rare cases (for example, internal errors during resource persistence), push may be prevented regardless of the annotation +{{% /alert %}} ## Rollback Mechanism @@ -490,57 +474,40 @@ When a draft is created, the engine sets up a rollback handler that will be invo 4. **Captures references** to the draft and repository for later use The rollback handler is a closure that captures the necessary context and can be invoked automatically when errors occur during the operation. - -**Rollback handler characteristics:** -- **Closure**: Captures draft and repository references -- **Best effort**: Logs warning if cleanup fails -- **Non-blocking**: Doesn't prevent error return -- **Automatic**: Invoked on any operation error +It captures draft and repository references and logs a warning if a cleanup fails. However, it is non-blocking, meaning it does not prevent error return. ### Rollback Triggers -**When rollback invoked:** +**Rollback is invoked for:** - Task application errors - Lifecycle update errors - Validation errors during operation - Any error after draft creation -**When rollback NOT invoked:** +**Rollback is NOT invoked for:** - Draft closure errors (would likely fail again) - Errors before draft creation (nothing to clean up) - Metadata update errors (draft already closed) ### Rollback Limitations -**Rollback may fail:** -- Repository connection lost -- Permission errors -- Draft already closed -- Repository in inconsistent state +The rollback may fail if the repository connection is lost, due to permission errors, +if the draft is already closed, or if the repository is in an inconsistent state. -**Failure handling:** -- Warning logged with error details -- Original operation error still returned -- Draft may remain in repository -- Manual cleanup may be required -- Repository garbage collection may clean up eventually +In case of failure, the warning is logged with error details, but the original operation error is still returned. +The draft may remain in the repository, which means that manual cleanup may be required. However, the repository +garbage collection may clean up eventually. ### Rollback vs Transaction -**Not a true transaction:** -- No two-phase commit -- No distributed transaction support -- Best-effort cleanup only +This is not classified as true transaction as there is no two-phase commit and no distributed +transaction support. It is best-effort cleanup only. -**Why not a transaction:** -- Git doesn't support transactions -- Repository operations not transactional -- Rollback is cleanup, not undo +Transactions are not used as Git does not support them and repository operations are not transactional. +Additionally, rollback is a cleanup, not an undo. -**Atomicity guarantee:** -- Either package revision created/updated or not -- No partial package revisions visible to clients -- Draft isolation prevents intermediate state visibility +Atomicity is guaranteed. Package revisions are either created/updated or not. There is no partial package +revisions visible to clients, as draft isolation prevents intermediate state visibility. ## Draft Lifecycle @@ -579,11 +546,9 @@ Drafts have a specific lifecycle within the workflow: ### Draft Isolation -**Isolation guarantees:** -- Draft changes not visible to other operations -- Draft workspace separate from other revisions -- Multiple drafts can exist simultaneously (different workspaces) -- Draft closure is atomic (all or nothing) +Isolation guarantees that draft changes are not visible to other operations, since the draft workspace is +separate from other revisions. This means, that multiple drafts can exist simultaneously, in different workspaces. +Draft closure is atomic (all or nothing). **Concurrency:** - Multiple drafts for different packages: Fully concurrent @@ -596,42 +561,28 @@ The Engine optimizes the draft-commit workflow: ### Optimization Strategies -**Lazy draft creation:** -- Draft created only when needed -- Not created for metadata-only updates -- Not created for read operations +During lazy draft creation, a draft is created only when needed. Drafts are not created for metadata-only updates, +or for read operations. -**Early validation:** -- Validation before draft creation -- Fails fast without expensive operations -- Reduces rollback frequency +Early validation is performed before draft creation. It fails fast without expensive operations +and reduces rollback frequency. -**Efficient draft closure:** -- Single repository operation -- Atomic commit to Git -- Minimal overhead +Another optimization is efficient draft closure, which uses single repository operation, +atomic commit to Git and minimal overhead. ### Performance Characteristics -**Draft creation cost:** -- Allocates workspace in repository -- Copies resources (for updates) -- Relatively lightweight - -**Draft modification cost:** -- In-memory operations -- No repository access during modifications -- Task handler does actual work - -**Draft closure cost:** -- Git commit/tag creation -- Repository write operation -- Most expensive part of workflow - -**Rollback cost:** -- Draft closure + deletion -- Two repository operations -- Only on error path +The draft creation cost is relatively lightweight. The workspace is allocated in the repository and +the resources, for updates, are copied. + +Draft modification costs consist of in-memory operations. During modifications, the repository is not +accessed, but the task handler works. + +The most expensive part of the workflow is the draft closure. It contains Git commit and tag creation, +as well as repository write operations. + +Rollback costs only appear on error path. This consists of two repository operations, draft closure +and deletion. ## Error Handling diff --git a/docs/content/en/docs/5_architecture_and_components/engine/functionality/lifecycle-management.md b/docs/content/en/docs/5_architecture_and_components/engine/functionality/lifecycle-management.md index 278f84297..2743aca89 100644 --- a/docs/content/en/docs/5_architecture_and_components/engine/functionality/lifecycle-management.md +++ b/docs/content/en/docs/5_architecture_and_components/engine/functionality/lifecycle-management.md @@ -78,11 +78,9 @@ The lifecycle state machine defines four states that a package revision can be i The engine treats both Published and DeletionProposed states as "published" for lifecycle checks. A package revision is considered published if its lifecycle is either Published or DeletionProposed. -**Why DeletionProposed is considered published:** -- Maintains immutability during deletion process -- Prevents modifications to package revisions being removed -- Ensures consistency in dependency checks -- Preserves audit trail until final deletion +DeletionProposed is considered published because it maintains immutability during deletion process. This state prevents modifications +to package revisions that are being removed and ensures consistency in dependency checks. It also preserves audit trail until the +final deletion. ## State Transitions @@ -107,19 +105,16 @@ The lifecycle state machine enforces specific allowed transitions: [Actual Deletion] ``` -**Forward transitions:** -- Draft → Proposed (submit for review) -- Proposed → Published (approve) -- Published → DeletionProposed (mark for deletion) - -**Backward transitions:** -- Proposed → Draft (return for rework) -- DeletionProposed → Published (reject deletion) - -**Forbidden transitions:** -- Published → Draft (cannot unpublish) -- Published → Proposed (cannot unpublish) -- Any state → Draft (except from Proposed) +| Direction | Transition | Action | +|-----------|------------|--------| +| Forward | Draft → Proposed | Submit for review | +| Forward | Proposed → Published | Approve | +| Forward | Published → DeletionProposed | Mark for deletion | +| Backward | Proposed → Draft | Return for rework | +| Backward | DeletionProposed → Published | Reject deletion | +| Forbidden | Published → Draft | Cannot unpublish | +| Forbidden | Published → Proposed | Cannot unpublish | +| Forbidden | Any state → Draft (except from Proposed) | — | ### Transition Validation @@ -444,27 +439,19 @@ The lifecycle system maintains an audit trail of package revision evolution: ### Audit Fields -**PublishedBy:** -- Records user who published the package revision -- Set when lifecycle transitions to Published -- Extracted from Kubernetes request context -- Stored in PackageRevision status +The PublishedBy field records the user who published the package revision. It is set when +the lifecycle transitions to Published. It is extracted from Kubernetes request context and +stored in the PackageRevision status. -**PublishedAt:** -- Timestamp when package revision was published -- Set when lifecycle transitions to Published -- Stored in PackageRevision status -- Used for tracking approval timing +The PublishedAt field is used for tracking approval timing. The time when a package revision +is published is timestamped, and PublishedAt is set when the lifecycle transitions to Published. +It is stored in the PackageRevision status. -**Tasks:** -- Typically contains a single task indicating creation method (init, clone, edit, upgrade) -- Stored in PackageRevision spec +The Tasks field typically contains a single task indicating creation method (init, clone, edit, upgrade). +It is stored in PackageRevision spec. -**Resource Version:** -- Kubernetes resource version for optimistic locking -- Incremented on each update -- Used to prevent concurrent modification conflicts -- Managed by Kubernetes API server +The Resource Version is the Kubernetes resource version for optimistic locking, which is incremented on +each update. This field is used to prevent concurrent modification conflicts and is managed by Kubernetes API server. ### Audit Trail Flow @@ -498,22 +485,13 @@ Actual Deletion ### Tracking Benefits -**Compliance:** -- Who approved package revisions for production -- When package revisions were approved -- What changes were made - -**Debugging:** -- Package revision evolution history -- Task execution sequence -- Lifecycle transition timeline - -**Rollback:** -- Identify previous stable versions -- Understand changes between versions -- Revert to known-good states - -**Governance:** -- Enforce approval workflows -- Audit package revision modifications -- Track package lineage +One benefit of tracking is compliance: it shows who approved package revisions for production, when they were approved, +and what changes were made. + +Debugging is easier due to the package revision evolution history, the task execution sequence, and the lifecycle +transition timeline. + +With rollback, previous stable versions can be identified. It is easier to understand changes between versions, and +possible to revert to known-good states. + +Finally, governance is improved by enforcing approval workflows, auditing package revision modifications, and tracking package lineage. diff --git a/docs/content/en/docs/5_architecture_and_components/engine/functionality/task-coordination.md b/docs/content/en/docs/5_architecture_and_components/engine/functionality/task-coordination.md index bdf22461f..1af4c86d2 100644 --- a/docs/content/en/docs/5_architecture_and_components/engine/functionality/task-coordination.md +++ b/docs/content/en/docs/5_architecture_and_components/engine/functionality/task-coordination.md @@ -58,17 +58,11 @@ UpdatePackageResources ─────> DoPRResourceMutations 2. **DoPRMutations**: Apply mutations during package revision update 3. **DoPRResourceMutations**: Apply resource mutations during resource update -**Engine responsibilities:** -- Determine when to invoke task handler -- Provide draft workspace for modifications -- Handle errors and rollback -- Manage lifecycle transitions - -**Task Handler responsibilities:** -- Execute task transformations -- Modify draft resources -- Apply builtin functions -- Return results or errors +The Engine is responsible for determining when to invoke task handler, providing draft workspace for modifications, +handling errors and rollback, and managing lifecycle transitions. + +The Task Handler is responsible for executing task transformations, modifying draft resources, applying builtin functions, +and returning results or errors. ## ApplyTask - Creation Task Execution @@ -117,39 +111,29 @@ Init Clone Edit Upgrade ### ApplyTask Parameters -**Draft:** -- Mutable workspace for modifications -- Provides UpdateResources, GetResources methods -- Isolated from other revisions +Draft is a mutable workspace for modifications. It provides UpdateResources, GetResources methods, and it +is isolated from other revisions. -**Repository Object:** -- Repository CR specification -- Used for context (repository name, namespace) -- Passed to task implementations +Repository Object is the Repository CR specification. It is used for context (repository name, namespace). +This parameter is passed to task implementations. -**PackageRevision:** -- Contains task specification in Spec.Tasks -- First task in list executed -- Task type determines which implementation runs +PackageRevision contains task specification in Spec.Tasks. The first task in the list is executed. Task type +determines which implementation runs. -**Package Config:** -- Package path, name, workspace -- Upstream reference (for clone/upgrade) -- Additional metadata +Package Config contains package path, name, and workspace. This is an upstream reference for clone/upgrade. +Contains additional metadata. ### Task Execution -**Task Handler executes the task:** -- Task type determines which implementation runs (init, clone, edit, upgrade) -- Task handler modifies draft resources based on task logic -- Builtin functions applied after task execution -- Modified draft returned to Engine +Task type determines which implementation runs (init, clone, edit, upgrade). Task handler modifies draft resources based on task logic, +and builtin functions are applied after task execution. The modified drafts are returned to Engine. -**Task types:** -- **init**: Create new package from scratch -- **clone**: Copy package from upstream -- **edit**: Create new revision from existing package -- **upgrade**: Merge changes from new upstream version +| Task type | Action | +|-----------|--------| +| init | Create new package from scratch | +| clone | Copy package from upstream | +| edit | Create new revision from existing package | +| upgrade | Merge changes from new upstream version | **Handback to Engine:** - **Success**: Returns modified draft @@ -202,25 +186,15 @@ UpdatePackageRevision ### DoPRMutations Parameters -**Repository PackageRevision:** -- Current package revision from repository -- Provides context for mutations -- Not directly modified (draft is modified) +Repository PackageRevision is the current package revision from the repository. It provides context for mutations, +but is not directly modified, the draft is. -**Old PackageRevision:** -- Previous PackageRevision spec -- Used for comparison -- Identifies what changed +Old PackageRevision is the previous PackageRevision spec. It is used for comparison, as it identifies what changed. -**New PackageRevision:** -- Desired PackageRevision spec -- Contains new tasks to apply -- Target state +New PackageRevision is the desired PackageRevision spec. It contains new tasks to apply to reach the target state. -**Draft:** -- Mutable workspace for modifications -- Already contains current package content -- Modified by new tasks +Draft is a mutable workspace for modifications. It already contains the current package content and it is modified by +new tasks. ### Task Comparison @@ -235,16 +209,11 @@ Compare Task Lists Apply New Tasks ``` -**Comparison logic:** -- Tasks are append-only (never removed) -- Compare task list lengths -- New tasks are those beyond old list length -- Apply new tasks in order +**Comparison logic:** Tasks are append-only (never removed) and task list lengths are compared. New tasks are those +beyond old list length. New tasks are applied in order. -**Task application:** -- Each new task executed sequentially -- Draft modified by each task -- Errors stop processing and return +**Task application:** Each new task is executed sequentially. The Draft is modified by each task. In case of an error, +the process stops and returns. ## DoPRResourceMutations - Resource Updates @@ -288,32 +257,21 @@ UpdatePackageResources ### DoPRResourceMutations Parameters -**PackageRevision:** -- Current package revision from repository -- Provides context for mutations -- Contains function pipeline configuration +PackageRevision is the current package revision from the repository, which provides context for mutations. +It contains the function pipeline configuration. -**Draft:** -- Mutable workspace for modifications -- Resources updated directly -- Modified by render task +Draft is a mutable workspace for modifications. The resources are updated directly and it is modified by +resource mutations (including the render task). -**Old PackageRevisionResources:** -- Previous resource content -- Used for comparison (not currently used) -- Audit trail +Old PackageRevisionResources is the previous resource content, which is used for comparison (not currently used). +It has an audit trail. -**New PackageRevisionResources:** -- Desired resource content -- Applied to draft -- Target state +New PackageRevisionResources is the desired resource content, which is applied to the draft. It is the target state. ### Render Execution -**Task Handler executes render:** -- Updates package resources in draft -- Executes render task (runs function pipeline) -- Returns RenderStatus with function results +The Task Handler updates package resources in draft and executes render task, which means running a function pipeline. +Once done, RenderStatus is returned with function results. **Handback to Engine:** - **RenderStatus**: Contains function execution results @@ -358,12 +316,10 @@ NewCaDEngine(opts...) - **gRPC Runtime**: For external function runner service - **Multi-Runtime**: Chains multiple runtimes together -**Runtime selection:** -- Configured at Porch server startup -- Passed to task handler during engine initialization -- Task handler uses runtime for function execution +Runtime selection is configured at Porch server startup. It is passed to task handler during engine initialization, +which uses runtime for function execution. -**For details on function runtime implementations, see [Function Runner]({{% relref "/docs/5_architecture_and_components/function-runner/_index.md" %}}).** +For details on function runtime implementations, see [Function Runner]({{% relref "/docs/5_architecture_and_components/function-runner/_index.md" %}}). ## Error Handling @@ -416,16 +372,10 @@ ApplyTask/DoPRMutations/DoPRResourceMutations ### Error Recovery -**Client retry:** -- Fix task configuration -- Resolve upstream references -- Fix resource syntax -- Retry operation +One option is client retry, which means fixing task configuration, resolving upstream references, fixing resource +syntax, and then retrying the operation. -**Automatic recovery:** -- Rollback on creation errors -- Draft cleanup -- No manual intervention needed +Automatic recovery includes rollback on creation errors, as well as draft cleanup. No manual intervention needed ## Task Coordination Patterns @@ -445,16 +395,11 @@ Task List: [init/clone/edit/upgrade] Return Success ``` -**Sequential execution:** -- Typically one task (init, clone, edit, or upgrade) -- Each task must succeed before next -- First error stops execution -- No parallel task execution +Tasks are executed sequentially. This means that typically one task (init, clone, edit, or upgrade) is executed at a time, +and each task must succeed before the next. The first error stops execution. No parallel task execution is allowed. -**Rationale:** -- Tasks may depend on previous tasks -- Simplifies error handling -- Maintains consistent state +This is needed because tasks may depend on previous tasks. This way, error handling is simplified and consistent state +is maintained. ### Task List Pattern @@ -466,9 +411,7 @@ Update: [clone/edit/upgrade] Update: [render] ``` -**Task list pattern:** -- Single persistent task indicating [init/clone/edit/upgrade] method -- Task history shows package origin +A single persistent task which indicates [init/clone/edit/upgrade] method. The task history shows package origin. ### Draft Isolation @@ -482,11 +425,8 @@ Package Revision A Package Revision B Independent Independent ``` -**Isolation guarantees:** -- Each draft has a separate context -- Task execution doesn't affect other drafts -- Concurrent task execution possible (different packages) -- No shared state between tasks +Isolation guarantees that each draft has a separate context, that task execution does not affect other drafts, and +that there is no shared state between tasks. With draft isolation, concurrent task execution is possible on different packages. ## Task Handler Interface @@ -511,20 +451,13 @@ The Engine interacts with Task Handler through a defined interface: ### Interface Characteristics -**Context-aware:** -- All methods accept context for cancellation -- Timeout and deadline support -- Tracing and logging context +The interface is context-aware. All methods accept context for cancellation. Timeout and deadline is supported. +Tracing and logging context is propagated as well. -**Draft-based:** -- All methods work with draft workspaces -- No direct repository modification -- Isolation and atomicity +It is draft-based, meaning that all methods work with draft workspaces and no direct repository modification +is performed. Includes isolation and atomicity. -**Error-returning:** -- Errors indicate task failure -- Engine handles rollback -- Clear error messages for debugging +Errors indicate task failure. The Engine handles rollback and gives clear error messages for debugging. ## Task Coordination Benefits @@ -532,23 +465,14 @@ The task coordination pattern provides several benefits: ### Separation of Concerns -**Engine:** -- Orchestrates workflow -- Manages drafts and lifecycle -- Handles errors and rollback -- Enforces business rules +The Engine orchestrates the workflow, manages drafts and lifecycle, handles errors and rollback, and +enforces business rules. -**Task Handler:** -- Implements task logic -- Transforms package content -- Executes functions -- Returns results +The Task Handler implements task logic, transforms package content, executes functions, and +returns results. -**Benefits:** -- Clear responsibilities -- Easier testing and maintenance -- Pluggable task implementations -- Independent evolution +The benefit of this separation is that both component have clear responsibilities. Testing and maintenance are easier, and +pluggable task implementations can be applied. Additionally, evolution of the components is independent. ### Extensibility @@ -566,14 +490,7 @@ The task coordination pattern provides several benefits: ### Testability -**Engine testing:** -- Mock task handler -- Test orchestration logic -- Test error handling -- Test rollback mechanism - -**Task Handler testing:** -- Mock draft interface -- Test task implementations -- Test function execution -- Independent of CaDEngine +You can test the engine with mock task handler. Orchestration logic, error handling, and rollback mechanism can be tested. + +You can test the Task Handler with mock draft interface. Task implementations and function execution can be tested. This is +independent of CaDEngine. diff --git a/docs/content/en/docs/5_architecture_and_components/engine/functionality/validation-business-rules.md b/docs/content/en/docs/5_architecture_and_components/engine/functionality/validation-business-rules.md index 69b54a945..99ba6e2f8 100644 --- a/docs/content/en/docs/5_architecture_and_components/engine/functionality/validation-business-rules.md +++ b/docs/content/en/docs/5_architecture_and_components/engine/functionality/validation-business-rules.md @@ -76,9 +76,9 @@ CreatePackageRevision - "unsupported lifecycle value: {value}" (for invalid values) **Rationale:** -- Packages must progress through Draft/Proposed before being published -- Prevents bypassing review/approval workflows -- Ensures all packages have a draft history + +Packages must progress through the Draft/Proposed states before being published. This prevents bypassing the review/approval +workflows and ensures all packages have a draft history. ### Update Lifecycle Validation @@ -108,9 +108,10 @@ UpdatePackageRevision - "invalid desired lifecycle value: {value}" **Rationale:** -- Published packages are immutable (content cannot change) -- Draft packages are mutable (work-in-progress) -- Lifecycle transitions must follow state machine rules (see [Lifecycle Management]({{% relref "/docs/5_architecture_and_components/engine/functionality/lifecycle-management.md" %}})) + +Published packages are immutable, meaning that their content cannot be changed. However, draft packages are mutable, since they are +work-in-progress. The lifecycle transitions must follow state machine rules (see +[Lifecycle Management]({{% relref "/docs/5_architecture_and_components/engine/functionality/lifecycle-management.md" %}})). ## Task Validation @@ -143,22 +144,23 @@ CreatePackageRevision - "task list must not contain more than one task" **Rationale:** -- Simplifies creation workflow (one operation at a time) -- Multiple tasks can be added later via updates -- Default init task provides sensible starting point + +Only allowing one operation simplifies the creation workflow. However, you can add multiple tasks later with updates. The default init +task provides s sensible starting point. ### Task Type Validation -**Valid task types:** -- **init**: Create new package from scratch -- **clone**: Copy package from upstream -- **edit**: Create new revision from existing package -- **upgrade**: Merge changes from new upstream version +| Valid task type | Description | +|-----------------|-------------------------------------------| +| init | Create new package from scratch | +| clone | Copy package from upstream | +| edit | Create new revision from existing package | +| upgrade | Merge changes from new upstream version | **Task-specific validation:** -- Each task type has additional validation rules -- Validation delegated to task-specific validators -- See sections below for details + +Each task type has additional validation rules and the validation is delegated to task-specific validators. +For more information, see the sections below. ## Workspace Name Uniqueness @@ -196,16 +198,15 @@ CreatePackageRevision - "package revision workspaceNames must be unique; package revision with name {name} in repo {repo} with workspaceName {workspace} already exists" **Rationale:** -- Workspace name used to generate Kubernetes object name -- Kubernetes object names must be unique within namespace -- Prevents naming conflicts and confusion + +The workspace name is used to generate the Kubernetes object name, which must be unique within the namespace. This +prevents naming conflicts and confusion. ### Workspace Name Validation -**Additional validation:** -- Workspace name must be valid Kubernetes name -- Combined with repository and package to form object name -- Format: `{repo}-{path}-{package}-{workspace}` +The workspace name must be a valid Kubernetes name. The repository and package is combined to create the object name. + +Format: `{repo}-{path}-{package}-{workspace}` ## Clone Task Validation @@ -247,9 +248,9 @@ The package name must be unique in the repository to avoid creating duplicate pa - "`clone` cannot create a new revision for package {package} that already exists in repo {repo}; make subsequent revisions using `copy`" **Rationale:** -- Clone is for creating NEW packages from upstream -- Existing packages should use `edit` or `copy` for new revisions -- Prevents accidental overwriting of existing packages + +Cloning is for creating **new** packages from upstream, which is why existing packages should use `edit` or `copy` for new revisions. +This prevents accidental overwriting of existing packages. ### Clone Constraint Check: Exclude Placeholder Package Revision @@ -265,21 +266,17 @@ The upstream package revision cannot be a placeholder package revision (identifi - "upstream revision may not be the placeholder package revision {repo}/{name}" **Rationale:** -- Placeholder package revisions represent the main/branch-HEAD state - - they are not fixed revisions, making them unsuitable for cloning -- Operations would fail in later lifecycle stages + +Placeholder package revisions represent the main/branch-HEAD state. This means that they are not fixed revisions, +making them unsuitable for cloning. These operations would fail in later lifecycle stages. ### Clone vs Edit/Copy -**When to use clone:** -- Creating a new package revision from upstream source -- Package doesn't exist in target repository -- First time bringing package into repository +Use **clone** when creating a new package revision from upstream source, when the package does not exist in target repository, +or when the package is brought into the repository for the first time. -**When to use edit/copy:** -- Creating new revision of existing package -- Package already exists in repository -- Iterating on existing package +Use **edit/copy** when creating a new revision of an existing package, when the package already exists in the repository, +or when you are iterating on an existing package. ## Edit Task Validation @@ -320,9 +317,9 @@ The new package revision must be from the same package as the source package rev - "source revision must be published" **Rationale:** -- Edit creates new revisions from existing packages in the same repository -- Placeholder package revisions represent unstable main branch state -- Only published revisions provide stable source for editing + +Edit creates new revisions from existing packages in the same repository. Placeholder package revisions represent +unstable main branch state. This means, that only published revisions provide stable source for editing. ### Edit Constraint Check: Exclude Placeholder Package Revision @@ -340,9 +337,9 @@ The source package revision cannot be a placeholder package revision (identified - "source revision may not be the placeholder package revision {repo}/{name}" **Rationale:** -- Placeholder package revisions are not fixed revisions -- Editing from unstable main/branch-HEAD state is not supported -- Prevents creating revisions from non-deterministic sources + +Placeholder package revisions are not fixed revisions, so editing from unstable main/branch-HEAD state is not supported. +This prevents creating revisions from non-deterministic sources. ## Upgrade Task Validation @@ -398,21 +395,19 @@ Upgrade Task Validation - "all source PackageRevisions of upgrade task must be published, {name} is not" **Rationale:** -- Upgrade performs three-way merge (old upstream, new upstream, local) -- Source revisions must be stable (published) for reliable merge -- Prevents upgrading from unstable draft versions + +Upgrade performs a three-way merge (old upstream, new upstream, local). Source revisions must be stable (published) for reliable merge. +This prevents upgrading from unstable draft versions. ### Upgrade Source Requirements -**Required sources:** -- **OldUpstream**: The package's current upstream version -- **NewUpstream**: The upstream version to which to upgrade -- **LocalPackageRevision**: The local package revision to upgrade +| Required sources | Description | +|----------------------|------------------------------------------| +| OldUpstream | The package's current upstream version | +| NewUpstream | The upstream version to which to upgrade | +| LocalPackageRevision | The local package revision to upgrade | -**All sources must be:** -- Published lifecycle state -- Accessible in repository -- Valid package revisions +All sources must be in published lifecycle state, accessible in the repository and valid package revisions. ### Placeholder Package Revision Check @@ -434,11 +429,10 @@ Neither the target upstream package revision (the new revision being upgraded to - "the placeholder package revision {repo}/{name} may not be upgraded" **Rationale:** -- Upgrade performs three-way merge requiring stable source revisions -- Placeholder package revisions represent unstable main branch state -- Using placeholder revisions would produce non-deterministic upgrade results -- Prevents upgrading to non-fixed revision states - - Validation on clone and edit operations precludes possibility of old upstream being a placeholder + +Upgrade performs a three-way merge requiring stable source revisions, however, placeholder package revisions represent unstable main branch state. +This means, that using placeholder revisions would produce non-deterministic upgrade results. This prevents upgrading to non-fixed revision states. +Validation on clone and edit operations precludes the possibility of old upstream being a placeholder. ## Package Path Overlap Validation @@ -475,9 +469,8 @@ CreatePackageRevision (init/clone) 5. **Allow if no overlap** **Path overlap rules:** -- Package path cannot be parent of existing package -- Package path cannot be child of existing package -- Prevents ambiguous package boundaries + +Package path cannot be the parent or a child of an existing package. This prevents ambiguous package boundaries. **Example overlaps (rejected):** - New: `networking/vpc`, Existing: `networking/vpc/subnets` (parent) @@ -488,9 +481,9 @@ CreatePackageRevision (init/clone) - New: `apps/frontend`, Existing: `apps/backend` (siblings) **Rationale:** -- Prevents nested package structures -- Maintains clear package boundaries -- Avoids confusion about package ownership + +This prevents nested package structures and maintains clear package boundaries. Additionally, this way confusion +about package ownership are avoided. ## Optimistic Locking @@ -568,6 +561,7 @@ Success (v2) <──────────── Return Success ### Resource Version Management **Resource version characteristics:** + - Managed by Kubernetes API server - Opaque string (typically integer) - Incremented on each update @@ -612,9 +606,8 @@ Execute Operation - Input format and structure **Benefits:** -- Fail fast (before expensive operations) -- Clear error messages -- No side effects on failure + +It is fail fast (before expensive operations), gives clear error messages, and has no side effects on failure. ### Mid-Operation Validation @@ -641,9 +634,8 @@ Close Draft - Package path overlaps **Benefits:** -- Access to repository state -- Can check against existing data -- Rollback mechanism available + +It has access to the repository state, can check against existing data and has rollback mechanism available. ### Post-Operation Validation @@ -677,9 +669,8 @@ The Engine returns specific errors for validation failures: ### Error Messages **Error message format:** -- Clear description of what failed -- Context (package name, repository, etc.) -- Suggestion for resolution when applicable + +The type of failure is clearly described with context given (package name, repository, etc.). Suggestion for resolution when applicable is also given. **Examples:** - "cannot create a package revision with lifecycle value 'Final'" @@ -708,12 +699,6 @@ The Engine's validation system can be extended: ### Validation Configuration -**Currently:** -- Validation rules are hardcoded in Engine -- No configuration mechanism +Currently validation rules are hardcoded in the Engine. No configuration mechanism is available. -**Future possibilities:** -- Configurable validation rules -- Repository-specific policies -- Organization-wide policies -- Validation rule versioning +Future possibilities include configurable validation rules, repository-specific policies, organization-wide policies and validation rule versioning. diff --git a/docs/content/en/docs/5_architecture_and_components/engine/interactions.md b/docs/content/en/docs/5_architecture_and_components/engine/interactions.md index ade638a20..d0879d88c 100644 --- a/docs/content/en/docs/5_architecture_and_components/engine/interactions.md +++ b/docs/content/en/docs/5_architecture_and_components/engine/interactions.md @@ -65,12 +65,9 @@ The engine never directly manipulates Git repositories or storage - all operatio ### Cache Invalidation -The engine doesn't directly invalidate cache entries. Instead: - -- Cache monitors Repository CRs for changes -- Background sync jobs refresh repository state periodically -- Repository operations (create, update, delete) trigger cache updates automatically -- Engine operations are always performed on the latest cached state +The engine does not directly invalidate cache entries. Instead, the cache monitors Repository CRs for changes, and background sync jobs refresh +repository state periodically. Repository operations (create, update, delete) trigger cache updates automatically. Engine operations are always +performed on the latest cached state. ## Task Handler Invocation @@ -149,11 +146,8 @@ Update resources + render 4. Task handler executes render task (runs function pipeline) 5. Returns render status with function results -**Key interaction points:** -- Engine provides the draft workspace -- Task handler modifies resources in the draft -- Engine commits the draft after task completion -- Task handler has no direct repository access +The Engine provides the draft workspace. The Task handler modifies resources in the draft, after which the Engine commits +the draft. The Task handler has no direct repository access. ### Function Runtime Integration @@ -183,7 +177,8 @@ External Repository (GitHub, GitLab, etc.) ### Repository Interface -The engine works with the `Repository` interface which provides: +The engine works with the `Repository` interface which provides Package Revision Management, Package Management, +and Repository Metadata. **Package Revision Management:** - Draft creation and closure @@ -224,23 +219,17 @@ The engine uses a draft-based workflow for all modifications: ### Credential and Reference Resolution -The engine is configured with resolvers for repository access: +The engine is configured with resolvers for repository access. -**Credential Resolver:** -- Resolves authentication credentials for repositories -- Supports basic auth, bearer tokens, and GCP workload identity -- Passed to repository adapters through cache -- Engine doesn't handle credentials directly +The Credential Resolver resolves authentication credentials for repositories. It supports basic auth, +bearer tokens, and GCP workload identity. It is passed to repository adapters through cache. The engine +does not handle credentials directly. -**Reference Resolver:** -- Resolves package revision references (upstream package revisions) -- Used during clone and upgrade operations -- Enables cross-repository package revision operations +The Reference Resolver resolves package revision references (upstream package revisions), and it is +used during clone and upgrade operations. It is used to enable cross-repository package revision operations. -**User Info Provider:** -- Provides authenticated user information -- Used for audit trails (PublishedBy field) -- Extracted from Kubernetes API request context +The User Info Provider provides authenticated user information, which is used for audit trails (PublishedBy field). +This info is extracted from Kubernetes API request context. These resolvers are configured during engine initialization and passed to components that need them. @@ -265,11 +254,8 @@ API Server Watch Streams - **Modified**: Package revision updated (metadata or lifecycle) - **Deleted**: Package revision removed -**When notifications are sent:** -- After successful package revision creation -- After package revision updates (including lifecycle transitions) -- After metadata-only updates on published packages -- Not sent on failures or during draft operations +Notifications are sent after successful package revision creation, after package revision updates (including lifecycle transitions), +and after metadata-only updates on published packages. Notifications are not sent on failures or during draft operations. ### Watch Stream Support @@ -281,22 +267,15 @@ The watcher manager enables Kubernetes watch API support: 4. **Event delivery**: Watcher manager delivers to matching watchers 5. **Client receives**: Watch event sent to client -**Filter support:** -- Repository name and namespace -- Package name and path -- Workspace name -- Lifecycle state -- Label selectors +You can filter for repository name and namespace, package name and path, workspace name, lifecycle state, and +label selectors. Only watchers with matching filters receive notifications. ### Watcher Lifecycle -Watchers are automatically cleaned up: - -- When client context is cancelled (connection closed) -- When watcher callback returns false (stop watching) -- Watcher manager periodically removes finished watchers -- No manual cleanup required from engine +Watchers are automatically cleaned up when client context is cancelled (connection closed) and when watcher callback +returns false (stop watching). The watcher manager periodically removes finished watchers. No manual cleanup is required +from engine. The engine simply notifies changes and the watcher manager handles delivery and cleanup.