ctrl-rs wip #488
Draft
ctrl-rs wip #488
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
…troller to match current remote controller Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Split out controller into core.rs and lib.rs. core.rs should be pure Rust and not depend on Python at all. * Update cargo.toml with two features - one that turns on pyo3/auto-initialize for building Rust crates (rlib), which allows us to cargo run binary files, and one that turns on pyo3/extension-module, which lets the manylinux image build wheels. * Temporarily adding in local flyte v2 idl - will remove next pr. * Update Phase -> ActionPhase. * Move the test functions in the Python BaseController class into separate test binaries now that we have rlib. (can delete these in the future) * Add an informer cache to support multiple root runs. * Hook up finalize parent action, which exposed the fact that Action::merge_from_submit was not being called, added. * Move controller errors out to a new file. * Update readme for setup docs a bit. --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Merged main into ctrl-rs. This PR is needed to get things working again. --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Update informer errors to use its own errors. * Add watch_for_errors. * Store the singular controller worker into a field so it can be raced in watch_for_errors. * Also race the informer error channel. * Some cleanup: remove unused fields from core base controller. * Adding a timeout for now to thread.join in _r_controller's stop. Otherwise sync tasks were hanging. needs further investigation. --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
- Add `make dev-rs-dist` to run `make build-wheels`, `make dist`, build image, and install `flyte_controller_base` locally - Stop `_submit_loop` in rust controller correctly when running sync task - Remove color code in remote logging (see following image) <img width="1186" height="470" alt="image" src="https://github.com/user-attachments/assets/be0840a8-e77d-46a9-9270-ab79de95bf15" /> --------- Signed-off-by: machichima <nary12321@gmail.com>
Reorganize module imports across several files to use grouped and brace-formatted use statements for better readability and consistency. Add a Makefile fmt target that enforces use of the nightly toolchain and runs `cargo +nightly fmt --all` to standardize formatting in CI and local development. These changes improve code consistency, maintainability, and make it easier to apply automatic formatting across the repository. Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Add Makefile targets for linting and autofix: - lint: run cargo clippy - lint-fix: run cargo clippy --fix Update CI workflow to run the Rust lint job: - add rs-lint job that checks out code, ensures toolchain, and runs make -C rs_controller lint Fix all clippy warnings. Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Add worker pool for rust controller Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Add an env var `_F_USE_RUST_CONTROLLER` that selects the rust based controller library. This will get recursively added also. Will not expose through a CLI switch yet... too experimental. Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Revert group data change Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Save action to informer action cache upon submit if not found in informer's action cache * `get_action` should call informer get or create, not just get, and return None rather than an error if informer doesn't have the action * Update flyteidl2 dep to match main project. --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.