[minor] Automated Job cleanup when auto_delete: false is set #257
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
relax-files identified by path relative to root dir
…a file, replacing need for the relax-list
This reverts commit bb1a113.
tomklapiscak
changed the title
(includes ansible-devops and python-devops fixes for 6.0.0 release)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
https://jsw.ibm.com/browse/MASCORE-5637
Introduces a mechanism to perform automated cleanup of orphaned Job resources when
auto_delete: falseis set.This PR addresses this by assigning a
mas.ibm.com/job-cleanup-grouplabel to Jobs. Any Job resources with the same value for this label, and in the same namespace are considered to be in the same cleanup group. A CronJob running in the cluster will delete all but the most recent (according to itscreationTimestamp) Jobs in each cleanup group.Validation has been added to the
verify-job-definitions.shscript (used as a pre-commit hook and in thelintGithub action) to ensure all relevant Jobs are set up to work correctly with this mechanism.The cleanup logic itself (in the form of a Python script) run by the CronJob is implemented here: ibm-mas/python-devops#58
For safety, the CronJob is assigned a ServiceAccount that can only list and delete Job resources (so it can never delete any other type of resource). Furthermore, the logic ensures that only Job resources with the
mas.ibm.com/job-cleanup-grouplabel can be deleted.This PR also bumps all Jobs to use the latest CLI version (
13.5.0->13.6.0)TODO
Testing
JobCleaner CronJob configured to run once a minute (as a stress test) during an fvtsaas deploy + test. Nothing untoward seen:
Subsequently updated cli image tag in fvtsaas with
auto_delete: falseset, temporarily suspend JobCleaner (to generate a large number of orphaned jobs). As expected, lots of orphaned Job resources left behind, e.g.:suite
sls
db2-db
syncjobs
Re-enabled JobCleaner; everything worked as expected:
suite
sls
db2-db
syncjobs
pre-commit hook:
