This repository was archived by the owner on Dec 15, 2021. It is now read-only.
Bug fixes on 0.2.2 + adding standard sql support + documentation#71
Open
lverzosa wants to merge 11 commits intospotify:masterfrom
Open
Bug fixes on 0.2.2 + adding standard sql support + documentation#71lverzosa wants to merge 11 commits intospotify:masterfrom
lverzosa wants to merge 11 commits intospotify:masterfrom
Conversation
* Updating versions * Revert "Update README.md" This reverts commit 55c75ec. * Revert "Added support for Spark 2.4.0 and Scala 2.12. Adjusted code slightly since Databricks Avro library moved into Spark with small syntax changes. (spotify#69)" This reverts commit ee9136b. * Revert "Update README.md (spotify#68)" This reverts commit a2d5a63. * Revert "Create daily partitioned table when necessary (spotify#67)" This reverts commit f5b4815. * Revert "Add maintenance mode notice" This reverts commit ab8fd9c. * Revert "Enable user to specify write options, hinting about issues with nested records (spotify#59)" This reverts commit 7f65455. * Revert "Configuration parameter for intermediate dataset id (spotify#63)" This reverts commit a0ecb3f. * Revert "Make a defensive copy to stop polution of sparkconf (spotify#60)" This reverts commit ffcd9a6. * Revert "Support setting query job priority (spotify#54)" This reverts commit 07e012a. * Fixing GoogleCredentials to use supplied file
Feature/revert to 2 2 (#1)
Adding flag for standard sql.
Adding example for standard SQL query
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi everybody,
LMK if you need more info, but effectively I just made changes to 0.2.2. There were a lot of library conflicts with the latest versions of the libraries since then so I've reverted those changes back.
In this PR:
Revert libraries to 0.2.2 version and scala back down to 2.11 (as 2.12 won't be supported until Spark 3.0)
Fixed GoogleCredential for bigQuerySelect to use credentials file (in Json) provided by setGcpJsonKeyFile() instead of GoogleCredential.getApplicationDefault()
Added flag for standard sql (while not breaking API for legacy SQL)
Documentation changes (removed reference to Dataproc since I believe google provides an up to date connector that works on Dataproc)
Lorenz