Abstract json processing differences between duckdb and postgres #51
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brycekbargar
merged commit cb81953
into
library-data-platform:release-v4.0.0
4 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The plan to speed up json transformation is to do it all in SQL using the native json functionality. A proof of concept and some napkin math has this running around 10x faster. In order to not have to duplicate the transformation queries (which will be complicated) we're adding abstractions over the native functionality in duckdb and postgres. This will allow us to write one query that is agnostic to the underlying engine.
Duckdb was in particular hard to get the versioning correct with as they like to have incompatibilities between versions. I think the version check and setting is a reasonable way to keep it consistent but we'll see if it becomes unmaintainable.
Postgres compatibility has been tested from 13.22+.
There's no actual functionality changes here but the PR was already getting huge before starting to do the actual big refactor for the transformation.