Add Job workload support to CRUD benchmarking framework#1133
Draft
diamondpowell wants to merge 10 commits into
Draft
Add Job workload support to CRUD benchmarking framework#1133diamondpowell wants to merge 10 commits into
diamondpowell wants to merge 10 commits into
Conversation
diamondpowell
force-pushed
the
dipowell/crud-jobs
branch
2 times, most recently
from
84f388c to
1da0a77
Compare
diamondpowell
force-pushed
the
dipowell/crud-jobs
branch
3 times, most recently
from
787e4d6 to
5a0978d
Compare
Add create_job() to NodePoolCRUD that deploys K8s Jobs onto node pools. Unlike deployments/statefulsets which run indefinitely, Jobs are run-to-completion workloads — success means the pod terminated cleanly (succeeded > 0), failure raises immediately (no self-healing). - Add 'jobs' subcommand to handle_workload_operations() in main.py with --number-of-jobs and --completions args - Add job.yml workload template with configurable completions and node affinity via label_selector - Add _check_job_condition and _is_job_condition_met to kubernetes_client.py — checks completion_time + succeeded count - Add wait_for_job_completed with 5-min timeout and 30s polling - Job kind support in apply/update/delete manifest methods
Add job execution step to the k8s CRUD engine pipeline between deployment and scale-down. Parameters (number_of_jobs, completions) flow from pipeline matrix → topology → engine step → main.py. - Add jobs script block to steps/engine/crud/k8s/execute.yml - Pass number_of_jobs and completions through topology execute-crud.yml - Jobs run after deployment, before scale-down + delete
Add comprehensive test coverage for create_job and job wait_for_condition: - test_create_job_success: single job completes successfully - test_create_job_failure: job fails to complete - test_create_job_partial_success: continues on individual failures - test_job_wait_for_condition: validates _check_job_condition and _is_job_condition_met for 'complete' and 'failed' states - test_wait_for_job_completed: timeout and polling behavior - Tests cover Job-specific semantics (succeeded count, completion_time, failed + no active pods)
- Extract _apply_job helper (matches _apply_deployment pattern) - Use os.path for default template path instead of hardcoded string - Use per-job labels to avoid selector collision - Remove redundant outer try/except - Use workload_common_parser for shared args (--count, --manifest-dir, etc.) - Add hasattr guard for cloud provider compatibility - Use args.count instead of args.number_of_jobs - Rename subcommand from 'jobs' to 'job' (matches K8s resource type) - Update pipeline YAML to use count parameter
- Wrap job pipeline step inside Azure cloud gate (matches deployment)
- Use ${MANIFEST_DIR:+--manifest-dir} conditional (matches deployment pattern)
diamondpowell
force-pushed
the
dipowell/crud-jobs
branch
from
8aed904 to
697d614
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Job workload support to the CRUD benchmarking framework — the third and final planned workload method. Unlike deployments/statefulsets which run indefinitely, Jobs are run-to-completion workloads: success means the pod terminated cleanly (
succeeded > 0), failure raises immediately.Branch cleanup note: Rebased and squashed for reviewability. Independent of StatefulSet PR (#1132).
Changes
modules/python/crud/workload_templates/job.ymlbatch/v1API withrestartPolicy: NeverJOB_COMPLETIONSplaceholder, no parallelism (defaults to match completions)modules/python/crud/azure/node_pool_crud.pycreate_job()— same loop pattern as other workloadscompletecondition instead ofavailable/readysince Jobs terminate after completionwait_for_pods_ready— pods exit after job finishesmodules/python/crud/main.pyjobssubparser with--node-pool-name,--number-of-jobs,--completions,--manifest-direlif command == "jobs"routing inhandle_workload_operationsmodules/python/clients/kubernetes_client.py_check_job_conditionand_is_job_condition_met— checkscompletion_time+succeededcountwait_for_job_completedwith 5-min timeout and 30s pollingapply_manifest,update_manifest,delete_manifeststeps/engine/crud/k8s/execute.ymljobsscript block callingpython3 main.py jobsnumber_of_jobsandcompletionsparameterssteps/topology/k8s-crud-gpu/execute-crud.ymlnumber_of_jobsandcompletionsthrough to engine stepmodules/python/clients/aks_client.pygpu_profiledriver to"None"for non-GPU node poolsTests
test_azure_node_pool_crud.py:test_create_job_successtest_create_job_failuretest_create_job_no_clienttest_create_job_partial_successtest_kubernetes_client.py:test_wait_for_condition_job_success— Job completes successfullytest_wait_for_condition_job_timeout— fails within timeouttest_wait_for_condition_job_not_found— not found, returns failureDependencies
Based on
test-refactor(PR #879) — must merge first. Independent of StatefulSet PR (#1132).