update etl configuration input to yml instead of environment variables#1795
update etl configuration input to yml instead of environment variables#1795adamkorynta wants to merge 6 commits into
Conversation
- file system staging update for projects, locations, and timeseries - add in REGI files to VCS - add in docker environment for running python tests - move etl calls to parent docker-compose file - let the source URL be optional, if not set only use staged filesystem resources
|
@dezidizon please review revisions to @RyanM-RMA's initial implementation |
adamkorynta
changed the title
dezidizon
left a comment
dezidizon
left a comment
There was a problem hiding this comment.
Performed code review and only found minor changes to capitalize several print out statements. This code is ready for merging even without implementing the requested updates.
Co-authored-by: dezidizon <47678178+dezidizon@users.noreply.github.com>
MikeNeilson
left a comment
MikeNeilson
left a comment
There was a problem hiding this comment.
Seems reasonable. Mostly commenting to track what I've already looked at.
|
|
||
| ## Configuration Overview | ||
|
|
||
| The main runtime configuration is stored in a YAML file, defaulting to `regi.yml` in the working directory. |
There was a problem hiding this comment.
Might want to name this something other than REGI. Not a major issue, but we might get less, I'll go with "side eye", if we use a more generic name.
There was a problem hiding this comment.
This likely doesn't need to be quite as detailed as this is (dealing with Oracle intricacies) https://github.com/HydrologicEngineeringCenter/cwms-database/blob/devops/flyway_migration/database-plugin/build.gradle, but it can help the flow of the build files to keep that more complex execution logic out of the main build.gradle files.
3 participants
Summary
Related Issue
Validation
Unit tests and docker compose in local dev environment
Checklist