Skip to content

Sync dev with v0.2.0 main #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sketchmind wants to merge 2 commits into from
Closed

Sync dev with v0.2.0 main #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3564886
Promote CLI self-description contracts
sketchmind Apr 19, 2026
94c06cd
Add workflow authoring and edit surfaces
sketchmind Apr 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,15 @@ All notable changes to this project will be documented in this file.
This project follows a simple public-release changelog until a stronger
versioning policy is needed.

## 0.2.0 - 2026-04-20

- Added typed task authoring schema discovery for workflow YAML creation.
- Added workflow and workflow-instance export commands for editable YAML.
- Added workflow and workflow-instance full-document edit flows alongside patch
edit templates.
- Improved schema, capabilities, README onboarding, and output-shape
documentation for agent and scripted use.

## 0.1.0 - 2026-04-15

- Generated-first DolphinScheduler REST contract runtime.
Expand Down
166 changes: 109 additions & 57 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,12 @@
# dolphinscheduler-cli

`dolphinscheduler-cli` is a generated-first, REST-only command-line interface
for Apache DolphinScheduler.
`dolphinscheduler-cli` provides `dsctl`, a command-line interface for Apache
DolphinScheduler.

The CLI is built around DolphinScheduler-native REST contracts and exposes them
through stable `dsctl` commands for configuration, authoring, runtime
inspection, and operational repair workflows. It does not use Py4J,
PyDolphinScheduler, or the Python gateway.
Use it for configuration, workflow authoring, runtime inspection, and
operational recovery through DolphinScheduler REST APIs.

This is an independent CLI project for Apache DolphinScheduler. It should not
be described as an official Apache DolphinScheduler distribution unless that
status changes explicitly.
This is an independent CLI project for Apache DolphinScheduler.

## Install

Expand All @@ -21,7 +17,7 @@ python -m pip install dolphinscheduler-cli
dsctl version
```

For isolated CLI usage:
With `pipx`:

```bash
pipx install dolphinscheduler-cli
Expand Down Expand Up @@ -58,61 +54,117 @@ You can also load connection settings from a dotenv-style file:
dsctl --env-file cluster.env context
```

## Quick Examples
## Quick Start

```bash
dsctl doctor
dsctl project list
dsctl use project etl-prod
dsctl workflow list
dsctl workflow get daily-etl
dsctl workflow run daily-etl
dsctl workflow-instance watch <workflow_instance_id>
dsctl workflow-instance digest <workflow_instance_id>
dsctl task-instance list --workflow-instance <workflow_instance_id>
dsctl task-instance log <task_instance_id> --raw
```

## Discover Commands

Start with `--help` for human-readable command entry points:

```bash
dsctl --help
dsctl workflow --help
dsctl workflow edit --help
```

Use `schema` for machine-readable arguments, options, choices, payload hints,
and output shape metadata:

```bash
dsctl schema --list-groups
dsctl schema --list-commands
dsctl schema --command workflow.edit
dsctl schema --command task-type.schema
```

Use `capabilities` for lightweight feature discovery:

```bash
dsctl capabilities --summary
dsctl capabilities --section authoring
```

## Workflow Authoring

Create workflow YAML from templates, lint it locally, then dry-run before
sending it to DolphinScheduler:

```bash
dsctl template task SHELL --raw
dsctl task-type schema SQL
dsctl template workflow --raw > workflow.yaml
dsctl lint workflow workflow.yaml
dsctl workflow create --file workflow.yaml --project etl-prod --dry-run
dsctl workflow create --file workflow.yaml --project etl-prod
```

Export an existing workflow, edit the YAML, and apply the full edited document:

```bash
dsctl workflow export daily-etl --project etl-prod > workflow.yaml
dsctl workflow edit daily-etl --project etl-prod --file workflow.yaml --dry-run
dsctl workflow edit daily-etl --project etl-prod --file workflow.yaml
```

For small changes, start from a patch template:

```bash
dsctl template workflow-patch --raw > patch.yaml
dsctl workflow edit daily-etl --project etl-prod --patch patch.yaml --dry-run
```

## Runtime Operations

```bash
dsctl workflow run daily-etl --project etl-prod
dsctl workflow run-task daily-etl --project etl-prod --task load
dsctl workflow-instance list --project etl-prod
dsctl workflow-instance digest <workflow_instance_id>
dsctl workflow-instance watch <workflow_instance_id>
dsctl workflow-instance recover-failed <workflow_instance_id>
dsctl task-instance list --workflow-instance <workflow_instance_id>
dsctl task-instance log <task_instance_id>
```

## Command Surface

Stable user-facing commands today:

- `dsctl version`
- `dsctl context`
- `dsctl doctor`
- `dsctl schema`
- `dsctl capabilities`
- `dsctl enum list ENUM`
- `dsctl lint workflow FILE`
- `dsctl task-type list`
- `dsctl env list|get|create|update|delete`
- `dsctl cluster list|get|create|update|delete`
- `dsctl datasource list|get|create|update|delete|test`
- `dsctl namespace list|get|available|create|delete`
- `dsctl resource list|view|upload|create|mkdir|download|delete`
- `dsctl queue list|get|create|update|delete`
- `dsctl worker-group list|get|create|update|delete`
- `dsctl task-group list|get|create|update|close|start`
- `dsctl task-group queue list|force-start|set-priority`
- `dsctl alert-plugin list|get|schema|create|update|delete|test`
- `dsctl alert-group list|get|create|update|delete`
- `dsctl tenant list|get|create|update|delete`
- `dsctl user list|get|create|update|delete`
- `dsctl user grant project|datasource|namespace`
- `dsctl user revoke project|datasource|namespace`
- `dsctl access-token list|get|create|update|delete|generate`
- `dsctl monitor health|server|database`
- `dsctl audit list|model-types|operation-types`
- `dsctl use project|workflow|--clear`
- `dsctl project list|get|create|update|delete`
- `dsctl project-parameter list|get|create|update|delete`
- `dsctl project-preference get|update|enable|disable`
- `dsctl project-worker-group list|set|clear`
- `dsctl schedule list|get|preview|explain|create|update|delete|online|offline`
- `dsctl template workflow|params|task`
- `dsctl workflow list|get|describe|digest|create|edit|online|offline|run|run-task|backfill|delete`
- `dsctl workflow lineage list|get|dependent-tasks`
- `dsctl workflow-instance list|get|parent|digest|update|watch|stop|rerun|recover-failed|execute-task`
- `dsctl task list|get|update`
- `dsctl task-instance list|get|watch|sub-workflow|log|force-success|savepoint|stop`
dsctl task-instance log <task_instance_id> --raw
```

Export a workflow instance before editing runtime task definitions:

```bash
dsctl workflow-instance export <workflow_instance_id> > instance.yaml
dsctl workflow-instance edit <workflow_instance_id> --file instance.yaml --dry-run
```

## Output

Commands return a stable JSON envelope by default. Use global output options
before the command group when a table or pipeline-oriented view is more useful:

```bash
dsctl --output-format table workflow-instance list --project etl-prod
dsctl --output-format tsv --columns id,name,state task-instance list --workflow-instance <workflow_instance_id>
dsctl --columns id,name,state workflow-instance list --project etl-prod
dsctl --output-format tsv --columns '*' task-instance list --workflow-instance <workflow_instance_id>
```

`--columns '*'` selects all top-level row fields. Quote `*` so the shell does
not expand it as a filesystem glob.

## Project Principles

- REST-only integration with DolphinScheduler APIs.
- Generated-first contracts for DS-facing request and response shapes.
- Stable command names, output envelopes, and error types for scripts and
agents.

## Documentation

Expand Down
16 changes: 9 additions & 7 deletions docs/development/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,11 @@ The current stable CLI surface is:
- `dsctl doctor`
- `dsctl schema`
- `dsctl capabilities`
- `dsctl enum names`
- `dsctl enum list ENUM`
- `dsctl lint workflow FILE`
- `dsctl task-type list`
- `dsctl env list|get|create|update|delete`
- `dsctl task-type list|get|schema`
- `dsctl environment list|get|create|update|delete`
- `dsctl cluster list|get|create|update|delete`
- `dsctl datasource list|get|create|update|delete|test`
- `dsctl namespace list|get|available|create|delete`
Expand All @@ -42,6 +43,7 @@ The current stable CLI surface is:
- `dsctl task-group list|get|create|update|close|start`
- `dsctl task-group queue list|force-start|set-priority`
- `dsctl alert-plugin list|get|schema|create|update|delete|test`
- `dsctl alert-plugin definition list`
- `dsctl alert-group list|get|create|update|delete`
- `dsctl tenant list|get|create|update|delete`
- `dsctl user list|get|create|update|delete`
Expand All @@ -56,10 +58,10 @@ The current stable CLI surface is:
- `dsctl project-preference get|update|enable|disable`
- `dsctl project-worker-group list|set|clear`
- `dsctl schedule list|get|preview|explain|create|update|delete|online|offline`
- `dsctl template workflow|params|task`
- `dsctl workflow list|get|describe|digest|create|edit|online|offline|run|run-task|backfill|delete`
- `dsctl template workflow|workflow-patch|workflow-instance-patch|params|environment|cluster|datasource|task`
- `dsctl workflow list|get|export|describe|digest|create|edit|online|offline|run|run-task|backfill|delete`
- `dsctl workflow lineage list|get|dependent-tasks`
- `dsctl workflow-instance list|get|parent|digest|update|watch|stop|rerun|recover-failed|execute-task`
- `dsctl workflow-instance list|get|export|parent|digest|edit|watch|stop|rerun|recover-failed|execute-task`
- `dsctl task list|get|update`
- `dsctl task-instance list|get|watch|sub-workflow|log|force-success|savepoint|stop`

Expand All @@ -72,7 +74,7 @@ The most useful split is:

| Plane | Core objects |
| --- | --- |
| Governance | user, access-token, tenant, queue, worker group, env, cluster, datasource, namespace, resource, alert plugin, alert group |
| Governance | user, access-token, tenant, queue, worker group, environment, cluster, datasource, namespace, resource, alert plugin, alert group |
| Project | project, project parameter, project preference, project worker-group, task group |
| Design | workflow, task, relation, workflow lineage, schedule |
| Runtime | command, workflow-instance, task-instance, audit log, logs, health |
Expand Down Expand Up @@ -279,7 +281,7 @@ Hard rules:

Current naming rules:

- governance and definition resources such as env, datasource, namespace,
- governance and definition resources such as environment, datasource, namespace,
queue, worker-group, tenant, user, project, workflow, and task are
name-first
- `resource` is path-first and uses the DS `fullName` path selector directly
Expand Down
36 changes: 19 additions & 17 deletions docs/development/live-testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,32 +291,33 @@ future.
| --- | --- | --- | --- |
| meta | `version`, `context` | No | local metadata and merged local context |
| schema | `schema`, `capabilities` | No | static CLI self-description |
| enum | `enum list` | No | generated enum metadata |
| enum | `enum names`, `enum list` | No | generated enum metadata |
| use | `use project|workflow|--clear` | No | local context persistence |
| lint | `lint workflow FILE` | No | local validation only |
| template | `template workflow|task` | No | local template rendering |
| template | `template workflow|workflow-patch|workflow-instance-patch|params|environment|cluster|datasource|task` | No | local template rendering |

### Meta And Diagnostics

| Surface | Commands | Default persona | Live required | Minimum live coverage |
| --- | --- | --- | --- | --- |
| doctor | `doctor` | `admin-bootstrap` and `etl-developer` | Yes | healthy preflight and broken-auth or broken-config variant |
| task-type | `task-type list` | `etl-developer` | Yes | real task-type discovery and category projection |
| task-type | `task-type get|schema` | No profile needed | No | local authoring summary and schema contracts |
| monitor | `monitor health`, `monitor server`, `monitor database` | `admin-bootstrap` | Yes | remote health, server, and database payloads reachable and shaped |
| audit | `audit list`, `audit model-types`, `audit operation-types` | `admin-bootstrap` or delegated governance user | Yes | remote filter metadata plus list query with real payload shape |

### Governance Surfaces

| Surface | Commands | Default persona | Live required | Minimum live coverage |
| --- | --- | --- | --- | --- |
| env | `env list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip plus not-found or validation case |
| environment | `environment list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip plus not-found or validation case |
| cluster | `cluster list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip plus delete cleanup |
| datasource | `datasource list|get|create|update|delete|test` | delegated governance user | Yes | CRUD round-trip and real connectivity test where the capability exists |
| namespace | `namespace list|get|available|create|delete` | delegated governance user | Yes | list and availability plus create/delete round-trip |
| resource | `resource list|view|upload|create|mkdir|download|delete` | `etl-developer` | Yes | file and directory lifecycle with content round-trip |
| queue | `queue list|get|create|update|delete` | `admin-bootstrap` or delegated governance user | Yes | CRUD round-trip plus permission boundary |
| worker-group | `worker-group list|get|create|update|delete` | `admin-bootstrap` or delegated governance user | Yes | CRUD round-trip plus selector correctness |
| alert-plugin | `alert-plugin list|get|schema|create|update|delete|test` | `admin-bootstrap` or delegated governance user | Yes | schema/read paths plus create/update/delete and plugin test where installed |
| alert-plugin | `alert-plugin list|get|definition list|schema|create|update|delete|test` | `admin-bootstrap` or delegated governance user | Yes | definition/schema/read paths plus create/update/delete and plugin test where installed |
| alert-group | `alert-group list|get|create|update|delete` | delegated governance user | Yes | CRUD round-trip and referenceable group payload shape |
| tenant | `tenant list|get|create|update|delete` | `admin-bootstrap` | Yes | CRUD round-trip plus non-admin denial case |
| user | `user list|get|create|update|delete|grant project|datasource|namespace|revoke project|datasource|namespace` | `admin-bootstrap` | Yes | CRUD plus grant/revoke effect and non-admin denial case |
Expand All @@ -338,7 +339,7 @@ future.
| workflow | `workflow list|get|describe|digest|create|edit|online|offline|run|run-task|backfill` | `etl-developer` | Yes | authoring, read-back, release-state transition, run, task-scoped run, backfill dry-run, and dry-run consistency |
| task | `task list|get|update` | `etl-developer` | Yes | live task projection and safe update reflected by later reads |
| schedule | `schedule list|get|preview|explain|create|update|delete|online|offline` | `etl-developer` | Yes | preview and explain aligned with create/update, and online schedules produce real workflow instances |
| workflow-instance | `workflow-instance list|get|parent|digest|update|watch|stop|rerun|recover-failed|execute-task` | `etl-developer` | Yes | instance lifecycle transitions, finished-instance DAG update semantics, runtime control, and parent/sub-workflow relation reads under real runtime conditions |
| workflow-instance | `workflow-instance list|get|parent|digest|edit|watch|stop|rerun|recover-failed|execute-task` | `etl-developer` | Yes | instance lifecycle transitions, finished-instance DAG edit semantics, runtime control, and parent/sub-workflow relation reads under real runtime conditions |
| task-instance | `task-instance list|get|watch|sub-workflow|log|force-success|savepoint|stop` | `etl-developer` | Yes | instance inspection, child relation reads, log retrieval, and control actions under real runtime conditions |

### Notes On Capability-Gated Resources
Expand All @@ -347,7 +348,7 @@ Some resources still require live coverage, but only in a compatible cluster:

- datasource connection tests need a reachable backend
- alert-plugin tests need installed plugin instances or plugin backends
- namespace, resource, env, and cluster operations may depend on deployment
- namespace, resource, environment, and cluster operations may depend on deployment
topology and storage configuration

The rule is not “skip forever”. The rule is:
Expand Down Expand Up @@ -383,7 +384,7 @@ Current verified coverage includes:
namespace capability/error paths
- project surfaces: `task-type`, `project`, `project-parameter`,
`project-preference`, `project-worker-group`
- runtime-adjacent governance: `cluster`, `env`, `resource`
- runtime-adjacent governance: `cluster`, `environment`, `resource`
- workflow runtime surfaces: `workflow`, `task`, `workflow-instance`,
`task-instance`, parent/sub-workflow relation reads, finished-instance DAG
update with and without definition sync, schedule-triggered runtime,
Expand All @@ -403,11 +404,12 @@ live additions.
- `resource view` is not reliable through the DS view endpoint because the
upstream controller misuses the `limit` parameter. The adapter now reads the
download endpoint and applies the line window client-side.
- `workflow-instance` task inspection in DS 3.4.1 is backed by
`GET /projects/{projectCode}/workflow-instances/{id}/tasks`, and the useful
payload lives under the DS `dataList` envelope as
`{ "taskList": [...], "workflowInstanceState": ... }`. Do not assume a page
envelope on this path.
- `task-instance list` in DS 3.4.1 is backed by the project-scoped
`GET /projects/{projectCode}/task-instances` paging query. The CLI narrows
the common per-run inspection path by sending `workflowInstanceId`, and it
uses the same path for broader project-scoped runtime triage filters.
Workflow-definition filtering is intentionally not exposed here because the
upstream BATCH query does not reliably apply `workflowDefinitionName`.
- `workflow describe` returns one root sentinel relation with
`preTaskCode=0`. That row is part of the DS DAG encoding and should not be
confused with a user-authored dependency edge.
Expand Down Expand Up @@ -435,9 +437,9 @@ live additions.
- schedule create also needs the workflow definition to be `ONLINE`, and the
current cluster requires explicit `tenantCode` plus a valid nonzero
`environmentCode`.
- `alert-plugin schema Script` exposes only the DS plugin definition metadata,
not a rich parameter form. The live suite should treat that as the upstream
contract unless source confirms otherwise.
- `alert-plugin definition list` discovers supported plugin definitions, while
`alert-plugin schema PLUGIN` fetches the full DS UI parameter form for one
definition when the upstream detail endpoint exposes it.
- `alert-plugin test` against the current Script plugin returns DS result code
`110014` when no executable script backend is configured. This is a valid
cluster capability failure, not a transport error.
Expand All @@ -453,7 +455,7 @@ live additions.
that same instance back into `FAILURE` before `workflow-instance rerun`
returns it to `SUCCESS`. Live assertions should follow the real state machine
instead of assuming a no-op on finished instances.
- `workflow-instance update` in DS 3.4.1 is a finished-instance save path, not
- `workflow-instance edit` in DS 3.4.1 is a finished-instance save path, not
a live-definition edit path. The server requires final-state instances with
usable `dagData`; `syncDefine=false` keeps the current workflow definition
unchanged, while `syncDefine=true` also persists the saved DAG back onto the
Expand Down Expand Up @@ -491,7 +493,7 @@ suite should cover more than basic create/list/delete flows.
### 4. Runtime Lifecycle

- workflow run and backfill dry-run
- workflow-instance list/get/digest/update/watch
- workflow-instance list/get/digest/edit/watch
- task-instance list/get/log
- stop, rerun, recover-failed, execute-task

Expand Down
Loading
Loading