feat(output-mapping): AJDA-2946 store description only when table flag is system-managed - #550
Draft
zajca wants to merge 6 commits into
Draft
feat(output-mapping): AJDA-2946 store description only when table flag is system-managed#550zajca wants to merge 6 commits into
zajca wants to merge 6 commits into
Conversation
zajca
force-pushed
the
zajca/AJDA-2946-om-description-system-managed-flag
branch
from
July 27, 2026 16:15
6601416 to
efc7b0e
Compare
zajca
changed the base branch from
main
to
zajca/om-untyped-create-via-table-definition
July 27, 2026 16:15
zajca
force-pushed
the
zajca/AJDA-2946-om-description-system-managed-flag
branch
from
July 27, 2026 19:34
efc7b0e to
9586553
Compare
zajca
force-pushed
the
zajca/om-untyped-create-via-table-definition
branch
from
July 29, 2026 08:34
311291d to
707acc6
Compare
…g is system-managed Table and column descriptions are stored in the native Storage description field through the table-definition update endpoint, driven by the table-level isDescriptionSystemManaged flag added in DMD-1662: - table created by the run: the description is always stored (a fresh table is system-managed by the Storage default), once the load finished and the table surely exists - this also covers tables created by the load job itself - existing table: the description is stored only when the flag is set, so a description managed by the user is never overwritten The existing KBC.description metadata written under the component provider is kept unchanged - it stays as provenance and as a read fallback.
…writing KBC.description Review of the previous commit surfaced three defects, all verified against real Storage: - Storage rejects a table-definition patch that carries no effective change with 400 "No table definition changes were provided.", so a second run with an unchanged description killed the whole output mapping - for an existing table even before any data was loaded. The current descriptions are now read from the table definition and only real changes are sent; when nothing changed, no call is made at all. This also removes the per-run cost in the steady state. - The definition-update endpoint exists on Snowflake and BigQuery only, so on any other backend every run with a description ended with 422. Storing the description is now skipped with an info message there. - Writing the native description makes Storage mirror it into a KBC.description metadata row under the `storage` provider, so output mapping's own componentId row was a duplicate - and on a user-managed table it published a competing description, which broke the acceptance criterion. KBC.description is no longer written as component metadata; resolution and removal live in DescriptionHelper. Both call sites now share a single flag-gated entry point taking TableInfo, so the guard cannot be bypassed, and a ClientException >= 500 is propagated instead of being reported as a user error, consistently with the metadata path.
…end provider Synapse, Exasol and Teradata are no longer Storage backends - the legacy BucketBackend factory registers Snowflake only and Model_Buckets::availableBackends() lists snowflake, bigquery and postgres. Postgres is the one live bucket backend the table-definition update endpoint does not support, so it is what the gate guards.
Snowflake, BigQuery and Postgres are the only bucket backends Storage still offers, and Postgres is not something output mapping needs to support here, so the gate guarded a case that cannot occur. Removing it takes with it the backend list duplicated from the platform, TableInfo::getBucketBackend() and the fixtures that only existed to feed it.
Base automatically changed from
zajca/om-untyped-create-via-table-definition
to
main
July 29, 2026 09:12
zajca
force-pushed
the
zajca/AJDA-2946-om-description-system-managed-flag
branch
from
July 29, 2026 09:12
9586553 to
3534278
Compare
A table created by output mapping through the table-definition API now carries its table and column descriptions directly in the create payload, so no extra blocking table-definition update job runs after the load. This covers all three create variants - typed from `schema`, typed from legacy column metadata, and non-typed from manifest `columns`. The deferred path is left only for a table created by the load job itself (CreateAndLoadTableTask), which has no create payload to embed into. LoadTableTaskCreator::create() therefore returns a LoadTableTaskResult carrying the descriptions that could not be embedded, and LoadTableQueue applies just those once the load finished. FailedLoadTableDecider now ignores the KBC.description metadata row under the "storage" provider. A description in the create payload makes Storage write that row at create time, before any data is loaded, so it says nothing about the table having been used and must not block the drop of a failed empty table. A failed load also drops its pending description, so the "was not stored" warning stays reserved for descriptions lost by mistake.
A review of the description coverage found the create paths well covered but the failure path, the second run and BigQuery untested. Closing those: - a failed load of a freshly created table carrying a description in its create payload must still drop the table; a companion test pins the assumption the drop depends on - that Storage writes the mirrored KBC.description row under the "storage" provider - a table which existed before a failed load keeps its data and description - BigQuery, newly reachable since the backend gate was dropped, on both the create payload and the table-definition update - a second run over a table created by the load job, where the diff reads the stored value back from the table definition - typed tables in both native-types suites: update, user-managed table, and the description of a column added by the same run - descriptions coming from `table_metadata` / `metadata` KBC.description, not only from the dedicated `description` field - a description dropped from the configuration, or sent empty, keeps the stored value instead of clearing it - a description of a column the data does not have is reported and skipped - workspace staging testSaveTableAndColumnMetadata and testLoadTaskTableNotExists asserted the description only through the derived metadata rows, which can no longer tell the native field apart from a row written by output mapping itself; both now assert `definition.description` directly, and the OR-fallback helpers that hid the difference are gone. The main description test threads the test logger through so the user-visible messages are assertable, and pins the table as non-typed so the non-typed path cannot silently stop being exercised.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves AJDA-2946.
Why
Until now output mapping stored the description as
KBC.descriptionmetadata and had no way to tell whether it may touch the description at all — writing it on every run would silently overwrite a description a user had set in the UI.DMD-1662 (connection #7754) added the table-level
isDescriptionSystemManagedflag exactly for this decision. This PR makes output mapping act on it and store the description in the native first-classdescriptionfield, which is what Input Mapping now reads primarily (AJDA-2714).This supersedes #525, which solved the same problem by guessing from the presence of a
userproviderKBC.descriptionrow. The flag replaces the heuristic.Behaviour
LoadTableQueue::waitForAll(), after a successful loadStoragePreparer, before the loadisDescriptionSystemManagedistrue; otherwise it is discarded and aninfomessage is loggedBoth paths go through one flag-gated entry point (
TableDescriptionModifier::updateDescriptions()) taking the table detail, so the guard cannot be bypassed on either.Only real changes are sent: the current descriptions are read from the table definition and compared with what the configuration produced. When nothing changed, no API call is made at all — so a steady-state run costs zero extra calls.
Descriptions of newly created tables are applied after the load rather than embedded in the
createTableDefinitionpayload. That is deliberate for two reasons: it is the only way to also cover tables output mapping does not create through a table definition (untyped tables, and tables with neither a manifest norcolumns, created by the load job itself), and embedding would breakFailedLoadTableDecider::decideTableDelete()— a description supplied at create time makes Storage write a metadata row immediately, socount($metadata) === 0would no longer hold and empty tables would stop being dropped after a failed import.KBC.descriptionis no longer written as component metadataWriting the native description makes Storage mirror it into a
KBC.descriptionmetadata row under thestorageprovider — for the table and for every column (TableDefinitionUpdateService::persistTableDescription()andStorageColumnDefinitionSynchronizer::syncColumnDescriptions()). Output mapping's own componentId row was therefore a duplicate, and on a user-managed table it published a competing description while the user's value sat understorage— for anyone reading metadata, output mapping had overwritten the user's description. That failed the acceptance criterion, so the key is no longer written as component metadata. Resolution and stripping live inWriter/Helper/DescriptionHelper.A column whose only metadata was the description is still reported by
getColumnMetadata()with an empty metadata list, because the set of columns to add to an existing table is derived from those keys as well (TableStructureModifier);ColumnsMetadatanow skips such empty entries instead of sending an empty metadata write.BC note: consumers that read
KBC.descriptionand filter onprovider == componentIdwill stop seeing it. The row under thestorageprovider remains, so consumers matching on the key are unaffected — including the input mapping manifest, which does not filter by provider.Notes for review
TableInfo::isDescriptionSystemManaged()defaults totruewhen the field is missing from the table detail, matching the platform default (and the DMD-1662 migration, where existing rows read as system-managed).ClientExceptionwith a code >= 500 is propagated instead of being wrapped, so a Storage outage stays a retryable application error — consistent with the metadata path inLoadTableQueue.keboola/storage-api-clientbumped^18.5→^18.9forupdateTableDefinition().LoadTableQueuegot a new optional 4th constructor argument, so it stays backwards compatible.main-writer-tests-2suite so it does not run in two CI jobs in parallel against the same project (bucket name collision).Testing
tests/Writer/TableDescriptionWriterTest.phpcovers the acceptance criteria against real Storage: a created table gets its description, a system-managed one is updated on the next run, a user-managed one is left alone, and a repeated run with an unchanged description succeeds.No table definition changes were provided.for a patch with no effective change, so every second run would have killed output mapping — for an existing table before any data was loaded.