Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,16 @@ dsctl project list
dsctl use project etl-prod
dsctl workflow list
dsctl workflow get daily-etl
dsctl template datasource --type MYSQL
dsctl workflow run daily-etl --project etl-prod
dsctl workflow-instance digest <workflow_instance_id>
dsctl task-instance list --workflow-instance <workflow_instance_id>
dsctl task-instance list --project etl-prod --state FAILURE
dsctl task-instance log <task_instance_id>
dsctl --columns id,name,state workflow-instance list --project etl-prod
dsctl --output-format table workflow-instance list --project etl-prod
dsctl --output-format tsv --columns id,name,state task-instance list --workflow-instance <workflow_instance_id>
dsctl --output-format tsv --columns '*' task-instance list --workflow-instance <workflow_instance_id>
```

## Command Surface
Expand All @@ -80,10 +86,11 @@ Stable user-facing commands today:
- `dsctl doctor`
- `dsctl schema`
- `dsctl capabilities`
- `dsctl enum names`
- `dsctl enum list ENUM`
- `dsctl lint workflow FILE`
- `dsctl task-type list`
- `dsctl env list|get|create|update|delete`
- `dsctl environment list|get|create|update|delete`
- `dsctl cluster list|get|create|update|delete`
- `dsctl datasource list|get|create|update|delete|test`
- `dsctl namespace list|get|available|create|delete`
Expand All @@ -93,6 +100,7 @@ Stable user-facing commands today:
- `dsctl task-group list|get|create|update|close|start`
- `dsctl task-group queue list|force-start|set-priority`
- `dsctl alert-plugin list|get|schema|create|update|delete|test`
- `dsctl alert-plugin definition list`
- `dsctl alert-group list|get|create|update|delete`
- `dsctl tenant list|get|create|update|delete`
- `dsctl user list|get|create|update|delete`
Expand All @@ -107,7 +115,7 @@ Stable user-facing commands today:
- `dsctl project-preference get|update|enable|disable`
- `dsctl project-worker-group list|set|clear`
- `dsctl schedule list|get|preview|explain|create|update|delete|online|offline`
- `dsctl template workflow|params|task`
- `dsctl template workflow|params|environment|cluster|datasource|task`
- `dsctl workflow list|get|describe|digest|create|edit|online|offline|run|run-task|backfill|delete`
- `dsctl workflow lineage list|get|dependent-tasks`
- `dsctl workflow-instance list|get|parent|digest|update|watch|stop|rerun|recover-failed|execute-task`
Expand Down
10 changes: 6 additions & 4 deletions docs/development/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,11 @@ The current stable CLI surface is:
- `dsctl doctor`
- `dsctl schema`
- `dsctl capabilities`
- `dsctl enum names`
- `dsctl enum list ENUM`
- `dsctl lint workflow FILE`
- `dsctl task-type list`
- `dsctl env list|get|create|update|delete`
- `dsctl environment list|get|create|update|delete`
- `dsctl cluster list|get|create|update|delete`
- `dsctl datasource list|get|create|update|delete|test`
- `dsctl namespace list|get|available|create|delete`
Expand All @@ -42,6 +43,7 @@ The current stable CLI surface is:
- `dsctl task-group list|get|create|update|close|start`
- `dsctl task-group queue list|force-start|set-priority`
- `dsctl alert-plugin list|get|schema|create|update|delete|test`
- `dsctl alert-plugin definition list`
- `dsctl alert-group list|get|create|update|delete`
- `dsctl tenant list|get|create|update|delete`
- `dsctl user list|get|create|update|delete`
Expand All @@ -56,7 +58,7 @@ The current stable CLI surface is:
- `dsctl project-preference get|update|enable|disable`
- `dsctl project-worker-group list|set|clear`
- `dsctl schedule list|get|preview|explain|create|update|delete|online|offline`
- `dsctl template workflow|params|task`
- `dsctl template workflow|params|environment|cluster|datasource|task`
- `dsctl workflow list|get|describe|digest|create|edit|online|offline|run|run-task|backfill|delete`
- `dsctl workflow lineage list|get|dependent-tasks`
- `dsctl workflow-instance list|get|parent|digest|update|watch|stop|rerun|recover-failed|execute-task`
Expand All @@ -72,7 +74,7 @@ The most useful split is:

| Plane | Core objects |
| --- | --- |
| Governance | user, access-token, tenant, queue, worker group, env, cluster, datasource, namespace, resource, alert plugin, alert group |
| Governance | user, access-token, tenant, queue, worker group, environment, cluster, datasource, namespace, resource, alert plugin, alert group |
| Project | project, project parameter, project preference, project worker-group, task group |
| Design | workflow, task, relation, workflow lineage, schedule |
| Runtime | command, workflow-instance, task-instance, audit log, logs, health |
Expand Down Expand Up @@ -279,7 +281,7 @@ Hard rules:

Current naming rules:

- governance and definition resources such as env, datasource, namespace,
- governance and definition resources such as environment, datasource, namespace,
queue, worker-group, tenant, user, project, workflow, and task are
name-first
- `resource` is path-first and uses the DS `fullName` path selector directly
Expand Down
29 changes: 15 additions & 14 deletions docs/development/live-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,10 +291,10 @@ future.
| --- | --- | --- | --- |
| meta | `version`, `context` | No | local metadata and merged local context |
| schema | `schema`, `capabilities` | No | static CLI self-description |
| enum | `enum list` | No | generated enum metadata |
| enum | `enum names`, `enum list` | No | generated enum metadata |
| use | `use project|workflow|--clear` | No | local context persistence |
| lint | `lint workflow FILE` | No | local validation only |
| template | `template workflow|task` | No | local template rendering |
| template | `template workflow|params|environment|cluster|datasource|task` | No | local template rendering |

### Meta And Diagnostics

Expand All @@ -309,14 +309,14 @@ future.

| Surface | Commands | Default persona | Live required | Minimum live coverage |
| --- | --- | --- | --- | --- |
| env | `env list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip plus not-found or validation case |
| environment | `environment list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip plus not-found or validation case |
| cluster | `cluster list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip plus delete cleanup |
| datasource | `datasource list|get|create|update|delete|test` | delegated governance user | Yes | CRUD round-trip and real connectivity test where the capability exists |
| namespace | `namespace list|get|available|create|delete` | delegated governance user | Yes | list and availability plus create/delete round-trip |
| resource | `resource list|view|upload|create|mkdir|download|delete` | `etl-developer` | Yes | file and directory lifecycle with content round-trip |
| queue | `queue list|get|create|update|delete` | `admin-bootstrap` or delegated governance user | Yes | CRUD round-trip plus permission boundary |
| worker-group | `worker-group list|get|create|update|delete` | `admin-bootstrap` or delegated governance user | Yes | CRUD round-trip plus selector correctness |
| alert-plugin | `alert-plugin list|get|schema|create|update|delete|test` | `admin-bootstrap` or delegated governance user | Yes | schema/read paths plus create/update/delete and plugin test where installed |
| alert-plugin | `alert-plugin list|get|definition list|schema|create|update|delete|test` | `admin-bootstrap` or delegated governance user | Yes | definition/schema/read paths plus create/update/delete and plugin test where installed |
| alert-group | `alert-group list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip and referenceable group payload shape |
| tenant | `tenant list|get|create|update|delete` | `admin-bootstrap` | Yes | CRUD round-trip plus non-admin denial case |
| user | `user list|get|create|update|delete|grant project|datasource|namespace|revoke project|datasource|namespace` | `admin-bootstrap` | Yes | CRUD plus grant/revoke effect and non-admin denial case |
Expand Down Expand Up @@ -347,7 +347,7 @@ Some resources still require live coverage, but only in a compatible cluster:

- datasource connection tests need a reachable backend
- alert-plugin tests need installed plugin instances or plugin backends
- namespace, resource, env, and cluster operations may depend on deployment
- namespace, resource, environment, and cluster operations may depend on deployment
topology and storage configuration

The rule is not “skip forever”. The rule is:
Expand Down Expand Up @@ -383,7 +383,7 @@ Current verified coverage includes:
namespace capability/error paths
- project surfaces: `task-type`, `project`, `project-parameter`,
`project-preference`, `project-worker-group`
- runtime-adjacent governance: `cluster`, `env`, `resource`
- runtime-adjacent governance: `cluster`, `environment`, `resource`
- workflow runtime surfaces: `workflow`, `task`, `workflow-instance`,
`task-instance`, parent/sub-workflow relation reads, finished-instance DAG
update with and without definition sync, schedule-triggered runtime,
Expand All @@ -403,11 +403,12 @@ live additions.
- `resource view` is not reliable through the DS view endpoint because the
upstream controller misuses the `limit` parameter. The adapter now reads the
download endpoint and applies the line window client-side.
- `workflow-instance` task inspection in DS 3.4.1 is backed by
`GET /projects/{projectCode}/workflow-instances/{id}/tasks`, and the useful
payload lives under the DS `dataList` envelope as
`{ "taskList": [...], "workflowInstanceState": ... }`. Do not assume a page
envelope on this path.
- `task-instance list` in DS 3.4.1 is backed by the project-scoped
`GET /projects/{projectCode}/task-instances` paging query. The CLI narrows
the common per-run inspection path by sending `workflowInstanceId`, and it
uses the same path for broader project-scoped runtime triage filters.
Workflow-definition filtering is intentionally not exposed here because the
upstream BATCH query does not reliably apply `workflowDefinitionName`.
- `workflow describe` returns one root sentinel relation with
`preTaskCode=0`. That row is part of the DS DAG encoding and should not be
confused with a user-authored dependency edge.
Expand Down Expand Up @@ -435,9 +436,9 @@ live additions.
- schedule create also needs the workflow definition to be `ONLINE`, and the
current cluster requires explicit `tenantCode` plus a valid nonzero
`environmentCode`.
- `alert-plugin schema Script` exposes only the DS plugin definition metadata,
not a rich parameter form. The live suite should treat that as the upstream
contract unless source confirms otherwise.
- `alert-plugin definition list` discovers supported plugin definitions, while
`alert-plugin schema PLUGIN` fetches the full DS UI parameter form for one
definition when the upstream detail endpoint exposes it.
- `alert-plugin test` against the current Script plugin returns DS result code
`110014` when no executable script backend is configured. This is a valid
cluster capability failure, not a transport error.
Expand Down
14 changes: 8 additions & 6 deletions docs/development/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,8 @@ dsctl use --clear
- [x] `--dry-run` support
- [x] `tests/models/test_workflow_spec.py` — YAML parsing tests
- [x] `tests/services/test_workflow.py` — create workflow tests
- [x] `dsctl template workflow` and `dsctl template task SHELL|SQL|HTTP|...`
- [x] `dsctl template workflow`, `dsctl template environment`, `dsctl template cluster`, and
`dsctl template task SHELL|SQL|HTTP|...`
- [x] YAML `schedule:` block support during `workflow create`
- [x] extend task-type coverage for DS logical/compound nodes:
`SUB_WORKFLOW`, `DEPENDENT`, `SWITCH`, `CONDITIONS`
Expand Down Expand Up @@ -331,10 +332,10 @@ for production use.
- [x] `queue` — CRUD
- [x] `worker-group` — CRUD
- [x] `task-group` — lifecycle plus task-group queue list/force-start/priority
- [x] `alert-plugin` — list/get/schema/create/update/delete/test
- [x] `alert-plugin` — list/get/definition list/schema/create/update/delete/test
- [x] `alert-group` — CRUD
- [x] `tenant` — CRUD
- [x] `env` — environment CRUD
- [x] `environment` — environment CRUD
- [x] `cluster` — cluster CRUD
- [x] `monitor` — health, servers, database stats
- [x] `audit` — list, model-types, operation-types
Expand Down Expand Up @@ -363,7 +364,7 @@ surface.
- [x] `dsctl lint` for local workflow design-time checks
- [x] `dsctl doctor` for runtime and governance diagnostics
- [x] `dsctl schema` — JSON tool definition output for the current stable surface
- [x] `dsctl enum list <enum>` — enum value discovery
- [x] `dsctl enum names`, `dsctl enum list <enum>` — enum value discovery
- [x] `dsctl task-type list` — live DS task-type discovery with favourite flags
- [x] audit log inspection and audit filter metadata discovery
- [x] workflow lineage inspection and dependent-task discovery
Expand Down Expand Up @@ -450,7 +451,7 @@ live contract test.
- [x] live contract cases for:
`tenant`, `user`, `access-token`, `queue`, `worker-group`
- [x] live contract cases for:
`env`, `cluster`, `alert-group`
`environment`, `cluster`, `alert-group`
- [x] permission-boundary tests for admin-only mutations
- [x] grant/revoke live tests where cluster policy allows them

Expand All @@ -462,7 +463,8 @@ have admin-path live coverage plus at least one denial or boundary case.
- [x] `datasource` live tests in an environment with real reachable backends
- [x] `alert-plugin` live tests in an environment with installed plugin
backends
- [x] capability-gated `env`, `cluster`, `namespace`, and `resource` scenarios
- [x] capability-gated `environment`, `cluster`, `namespace`, and `resource`
scenarios
in compatible deployments where needed
- [x] any skipped suite records the missing capability explicitly

Expand Down
Loading