[V2] feat: Create Helm charts for flyte-core#7285
Closed
lowc1012 wants to merge 247 commits into
Closed
Conversation
* Add flyteidl and minimal publishing workflows Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * pretend version for buf python Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
* python version fix and use OIDC to authenticate crates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix python versioning Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Set version before setting up env Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * echo version Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Write to GITHUB_ENV Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Use env: Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf generate Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
* Mark go package as /v2 to be able to import both at the same time Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Upgrade to go 1.24 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Rename python, rust and npm to flyteidl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
…lyteorg#6610) * trying idl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update trace def Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * generate -> buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * build: re-gen Signed-off-by: machichima <nary12321@gmail.com> * fix: exclude python venv Signed-off-by: machichima <nary12321@gmail.com> * Add translator service Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf gen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * remove node execution service proto Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Breakup task service into a separate package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Help strings for make file targets Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update TaskLog with read bool Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trying the updated protovalidate plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * protovalidate with semver Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try claude :shrug Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update deps Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * build: remove generate validate for python Signed-off-by: machichima <nary12321@gmail.com> * build: use managed mode on buf.gen.yaml Signed-off-by: machichima <nary12321@gmail.com> * Update validate go plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update imagebuild Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix oneof buf lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add deps to generated python code Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Be explicit about the index Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updated idl Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Catch up to main Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Generate mocks for workflow package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * generate mocks for other packages Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf improvements Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * feat: generate mock for common Signed-off-by: machichima <nary12321@gmail.com> * use make gen in GH Actions Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Do not setup python env in generation CI Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * setup uv Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set working directory Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Try to reference files by path Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Explicitly move .venv to root Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try exclude in buf.yaml Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set UV_PROJECT_ENVIRONMENT Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * download tooling Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add which mockery Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com>
* Add Trigger APIs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update triggers Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
* Fix cargo build and add CI check Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint lib.rs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update Readme and contributing guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * signed? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build-crate target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix included ts files in npm Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Add device class to `GpuAccelerator` Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * make gen Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add HABANA_GAUDI as supported device class Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Clean up triggers api Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * make gen Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> --------- Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
* make state service apis unary Signed-off-by: Daniel Rammer <hamersaw@protonmail.com> * make buf && make gen Signed-off-by: Daniel Rammer <hamersaw@protonmail.com> --------- Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>
…g#6663) Signed-off-by: Alex Wu <c.alexwu@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Use Docker images locally and in CI Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix branch prefix tag Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * use github.head_ref instead Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix tag in comment Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * make local docker experience Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Faster python install? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wait for image build wf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Optimize image Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * cleanups Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * cleanup not needed commands Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * test changes Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trying with :v2 image Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix WF Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/check-generate.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update DOCKER_QUICK_START.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update README.md Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update CONTRIBUTING.md Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
* fix(runs): persist model CreatedAt in CreateAction for correct trace ordering Trace actions set CreatedAt to the span's start_time so that parents (which start before children but are recorded later) sort earlier in ListActions ORDER BY created_at ASC. The INSERT was omitting created_at, causing the DB to use CURRENT_TIMESTAMP (insert time) and reversing the parent/child order — leading to "parent node not found" errors in WatchActions when a child action was processed before its parent. Non-trace actions leave CreatedAt as the zero value and fall back to time.Now(), preserving existing behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add end time Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix sql sentence Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
…lyteorg#7190) Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Alex Wu <c.alexwu@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>
…yteorg#7213) Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
… numbering (flyteorg#7220) Signed-off-by: Kevin Su <pingsutw@apache.org>
* proto stubs Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * add demo cluster impl Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * impl Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * mocks Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * Cleaner Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * tests Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * dockerfile Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * move log_streamer to dataproxy Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * regen Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * regen Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * revert, restore, redo Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * revert Signed-off-by: Katrina Rogan <katroganGH@gmail.com> --------- Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
…K8s cascade (flyteorg#7225) Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>
…logKeys (flyteorg#7229) * [V2] Expand contextutils with org, service_name, cluster_id keys and logKeys The shared interceptor chain in Union Cloud sets context values using v1 contextutils.Key types, but v2 services (actions, leaseworker, leasor, events-proxy, executorv2) use the v2 logger which reads v2 contextutils.Key. Both are `type Key string` but are different Go types, so context.Value() lookups fail silently and all context-propagated log fields are missing from v2 service logs. Add OrganizationKey, ServiceNameKey, and ClusterNameKey constants to v2 contextutils, and expand logKeys with ProjectKey, DomainKey, and the three new keys so that GetLogFields() includes them in structured log output. This is one half of the fix — the other half is a context bridge interceptor in the cloud repo that copies v1 values onto v2 keys at request time. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * make gen Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai>
* add: dep Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restructure Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * wip Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restruct Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * move config Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * remove db.go Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * address comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…inistic (flyteorg#7231) * fix(runs): make TestWatchActionUpdates_OnlyStreamsTargetAction deterministic Start watcher before creating actions so the subscriber is registered when the creation NOTIFY arrives. Drain the creation notification before the real assertions to eliminate the race that caused spurious "unexpected update for action target" failures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(runs): replace sleep with deterministic subscriber check Replace time.Sleep with require.Eventually polling actionSubscribers to ensure the watcher is registered before creating actions. This removes all timing dependencies from the test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…org#7230) * fix(Makefile): preserve exit code in timed macro The timed macro previously lost the exit code of the wrapped command due to chaining with semicolons — make always saw the exit code from the final echo statement (0). Now the macro captures the command's exit code in a variable, prints the timing, then exits with the original code, ensuring CI targets like buf-lint properly fail when violations are detected. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(proto): resolve all buf lint violations - Remove unused import of google/api/annotations.proto from cluster/service.proto - Remove unused import of flyteidl2/task/common.proto from queue_service.proto - Fix package mismatch in settings protos: change package from flyteidl2.org to flyteidl2.settings and update all field option references accordingly - Remove unused buf dependency buf.build/grpc-ecosystem/grpc-gateway - Regenerate all generated files (Go, Python, Rust, TypeScript) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
…g#7175) * add: dep Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restructure Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * wip Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restruct Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * move config Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * remove db.go Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * address comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * impl: internal service Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * add tests Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * address comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Signed-off-by: Ryan Lo <wenchih@apache.org>
Member
|
@lowc1012 Can you target the main branch? |
Contributor
Author
|
@Sovietaced closed this PR, feel free to create a new one |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue
Closes #7203
Why are the changes needed?
Create Helm charts for flyte-core v2
What changes were proposed in this pull request?
How was this patch tested?
Labels
Please add one or more of the following labels to categorize your PR:
This is important to improve the readability of release notes.
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link
main