Skip to content

feat(output-mapping): AJDA-2946 store description only when table flag is system-managed - #550

Draft
zajca wants to merge 6 commits into
mainfrom
zajca/AJDA-2946-om-description-system-managed-flag
Draft

feat(output-mapping): AJDA-2946 store description only when table flag is system-managed#550
zajca wants to merge 6 commits into
mainfrom
zajca/AJDA-2946-om-description-system-managed-flag

Conversation

@zajca

@zajca zajca commented Jul 27, 2026

Copy link
Copy Markdown
Member

Resolves AJDA-2946.

Why

Until now output mapping stored the description as KBC.description metadata and had no way to tell whether it may touch the description at all — writing it on every run would silently overwrite a description a user had set in the UI.

DMD-1662 (connection #7754) added the table-level isDescriptionSystemManaged flag exactly for this decision. This PR makes output mapping act on it and store the description in the native first-class description field, which is what Input Mapping now reads primarily (AJDA-2714).

This supersedes #525, which solved the same problem by guessing from the presence of a user provider KBC.description row. The flag replaces the heuristic.

Behaviour

Situation Where What happens
Table is created by this run LoadTableQueue::waitForAll(), after a successful load description is stored — a fresh table is system-managed by the Storage default
Table already existed StoragePreparer, before the load description is stored only when isDescriptionSystemManaged is true; otherwise it is discarded and an info message is logged

Both paths go through one flag-gated entry point (TableDescriptionModifier::updateDescriptions()) taking the table detail, so the guard cannot be bypassed on either.

Only real changes are sent: the current descriptions are read from the table definition and compared with what the configuration produced. When nothing changed, no API call is made at all — so a steady-state run costs zero extra calls.

Descriptions of newly created tables are applied after the load rather than embedded in the createTableDefinition payload. That is deliberate for two reasons: it is the only way to also cover tables output mapping does not create through a table definition (untyped tables, and tables with neither a manifest nor columns, created by the load job itself), and embedding would break FailedLoadTableDecider::decideTableDelete() — a description supplied at create time makes Storage write a metadata row immediately, so count($metadata) === 0 would no longer hold and empty tables would stop being dropped after a failed import.

KBC.description is no longer written as component metadata

Writing the native description makes Storage mirror it into a KBC.description metadata row under the storage provider — for the table and for every column (TableDefinitionUpdateService::persistTableDescription() and StorageColumnDefinitionSynchronizer::syncColumnDescriptions()). Output mapping's own componentId row was therefore a duplicate, and on a user-managed table it published a competing description while the user's value sat under storage — for anyone reading metadata, output mapping had overwritten the user's description. That failed the acceptance criterion, so the key is no longer written as component metadata. Resolution and stripping live in Writer/Helper/DescriptionHelper.

A column whose only metadata was the description is still reported by getColumnMetadata() with an empty metadata list, because the set of columns to add to an existing table is derived from those keys as well (TableStructureModifier); ColumnsMetadata now skips such empty entries instead of sending an empty metadata write.

BC note: consumers that read KBC.description and filter on provider == componentId will stop seeing it. The row under the storage provider remains, so consumers matching on the key are unaffected — including the input mapping manifest, which does not filter by provider.

Notes for review

  • TableInfo::isDescriptionSystemManaged() defaults to true when the field is missing from the table detail, matching the platform default (and the DMD-1662 migration, where existing rows read as system-managed).
  • An empty description is treated as no description, so it never clears a value stored in Storage.
  • Column descriptions are filtered against the columns the table really has; skipped ones are logged as a warning rather than failing the request (Storage answers 404 for an unknown column and would reject the whole patch, table description included).
  • A ClientException with a code >= 500 is propagated instead of being wrapped, so a Storage outage stays a retryable application error — consistent with the metadata path in LoadTableQueue.
  • keboola/storage-api-client bumped ^18.5^18.9 for updateTableDefinition().
  • LoadTableQueue got a new optional 4th constructor argument, so it stays backwards compatible.
  • The new functional test is excluded from the main-writer-tests-2 suite so it does not run in two CI jobs in parallel against the same project (bucket name collision).

Testing

  • phpcs + phpstan (level max) clean; unit tests green.
  • Functional test tests/Writer/TableDescriptionWriterTest.php covers the acceptance criteria against real Storage: a created table gets its description, a system-managed one is updated on the next run, a user-managed one is left alone, and a repeated run with an unchanged description succeeds.
  • That last case is what caught the most serious defect of the first revision: Storage answers 400 No table definition changes were provided. for a patch with no effective change, so every second run would have killed output mapping — for an existing table before any data was loaded.

@linear-code

linear-code Bot commented Jul 27, 2026

Copy link
Copy Markdown

AJDA-2946

@zajca
zajca force-pushed the zajca/AJDA-2946-om-description-system-managed-flag branch from 6601416 to efc7b0e Compare July 27, 2026 16:15
@zajca
zajca changed the base branch from main to zajca/om-untyped-create-via-table-definition July 27, 2026 16:15
@zajca
zajca force-pushed the zajca/AJDA-2946-om-description-system-managed-flag branch from efc7b0e to 9586553 Compare July 27, 2026 19:34
@zajca
zajca force-pushed the zajca/om-untyped-create-via-table-definition branch from 311291d to 707acc6 Compare July 29, 2026 08:34
zajca added 4 commits July 29, 2026 10:40
…g is system-managed

Table and column descriptions are stored in the native Storage description
field through the table-definition update endpoint, driven by the table-level
isDescriptionSystemManaged flag added in DMD-1662:

- table created by the run: the description is always stored (a fresh table is
  system-managed by the Storage default), once the load finished and the table
  surely exists - this also covers tables created by the load job itself
- existing table: the description is stored only when the flag is set, so a
  description managed by the user is never overwritten

The existing KBC.description metadata written under the component provider is
kept unchanged - it stays as provenance and as a read fallback.
…writing KBC.description

Review of the previous commit surfaced three defects, all verified against real
Storage:

- Storage rejects a table-definition patch that carries no effective change with
  400 "No table definition changes were provided.", so a second run with an
  unchanged description killed the whole output mapping - for an existing table
  even before any data was loaded. The current descriptions are now read from the
  table definition and only real changes are sent; when nothing changed, no call
  is made at all. This also removes the per-run cost in the steady state.
- The definition-update endpoint exists on Snowflake and BigQuery only, so on any
  other backend every run with a description ended with 422. Storing the
  description is now skipped with an info message there.
- Writing the native description makes Storage mirror it into a KBC.description
  metadata row under the `storage` provider, so output mapping's own componentId
  row was a duplicate - and on a user-managed table it published a competing
  description, which broke the acceptance criterion. KBC.description is no longer
  written as component metadata; resolution and removal live in DescriptionHelper.

Both call sites now share a single flag-gated entry point taking TableInfo, so the
guard cannot be bypassed, and a ClientException >= 500 is propagated instead of
being reported as a user error, consistently with the metadata path.
…end provider

Synapse, Exasol and Teradata are no longer Storage backends - the legacy
BucketBackend factory registers Snowflake only and Model_Buckets::availableBackends()
lists snowflake, bigquery and postgres. Postgres is the one live bucket backend the
table-definition update endpoint does not support, so it is what the gate guards.
Snowflake, BigQuery and Postgres are the only bucket backends Storage still
offers, and Postgres is not something output mapping needs to support here, so
the gate guarded a case that cannot occur. Removing it takes with it the backend
list duplicated from the platform, TableInfo::getBucketBackend() and the fixtures
that only existed to feed it.
Base automatically changed from zajca/om-untyped-create-via-table-definition to main July 29, 2026 09:12
@zajca
zajca force-pushed the zajca/AJDA-2946-om-description-system-managed-flag branch from 9586553 to 3534278 Compare July 29, 2026 09:12
zajca added 2 commits July 29, 2026 15:02
A table created by output mapping through the table-definition API now carries
its table and column descriptions directly in the create payload, so no extra
blocking table-definition update job runs after the load. This covers all three
create variants - typed from `schema`, typed from legacy column metadata, and
non-typed from manifest `columns`.

The deferred path is left only for a table created by the load job itself
(CreateAndLoadTableTask), which has no create payload to embed into.
LoadTableTaskCreator::create() therefore returns a LoadTableTaskResult carrying
the descriptions that could not be embedded, and LoadTableQueue applies just
those once the load finished.

FailedLoadTableDecider now ignores the KBC.description metadata row under the
"storage" provider. A description in the create payload makes Storage write that
row at create time, before any data is loaded, so it says nothing about the table
having been used and must not block the drop of a failed empty table.

A failed load also drops its pending description, so the "was not stored"
warning stays reserved for descriptions lost by mistake.
A review of the description coverage found the create paths well covered but the
failure path, the second run and BigQuery untested. Closing those:

- a failed load of a freshly created table carrying a description in its create
  payload must still drop the table; a companion test pins the assumption the
  drop depends on - that Storage writes the mirrored KBC.description row under
  the "storage" provider
- a table which existed before a failed load keeps its data and description
- BigQuery, newly reachable since the backend gate was dropped, on both the
  create payload and the table-definition update
- a second run over a table created by the load job, where the diff reads the
  stored value back from the table definition
- typed tables in both native-types suites: update, user-managed table, and the
  description of a column added by the same run
- descriptions coming from `table_metadata` / `metadata` KBC.description, not
  only from the dedicated `description` field
- a description dropped from the configuration, or sent empty, keeps the stored
  value instead of clearing it
- a description of a column the data does not have is reported and skipped
- workspace staging

testSaveTableAndColumnMetadata and testLoadTaskTableNotExists asserted the
description only through the derived metadata rows, which can no longer tell the
native field apart from a row written by output mapping itself; both now assert
`definition.description` directly, and the OR-fallback helpers that hid the
difference are gone. The main description test threads the test logger through so
the user-visible messages are assertable, and pins the table as non-typed so the
non-typed path cannot silently stop being exercised.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant