DAOS-19092 test: Run config generate tests on VMD.#18647
Draft
phender wants to merge 16 commits into
Draft
Conversation
Updates `actions/setup-python` from 6.2.0 to 6.3.0 Signed-off-by: dependabot[bot] <support@github.com>
#18519) Add faults tag to recovery tests that use fault injection tool. Signed-off-by: Makito Kano <makito.kano@hpe.com>
Disable unit-testing.yml on PRs since we do not have self-hosted runners. Signed-off-by: Dalton Bohning <dalton.bohning@hpe.com>
Introduce the Go test suite for the ddb CLI layer, built on top of the build-tag CGo stub infrastructure introduced with commit 86f31a2: - Add test_helpers.go: newTestContext(t) resets all CGo stubs via resetDdbStubs() and returns a *DdbContext ready for use in tests. Test cases set per-function _Fn hook variables directly. - All test files carry the //go:build test_stubs tag so they only compile when the stub infrastructure is present. - TestDdb_Cmds: open (default, write_mode, db_path), feature (show, enable, disable), and dtx_aggr (mutual exclusion, cmt_time, cmt_date, path). - TestDdb_HelpCmds: --help output for the ls and open commands. - TestDdb_parseOpts / TestDdb_runDdb / TestDdb_runDdbCommandFile: CLI-level option parsing and runDdb() dispatch, including unknown-command detection for both command-line and command-file paths. - TestDdb_runFileCmds: verifies that a multi-line command file executes all commands in order. - TestDdb_newLogger: 6 sub-cases (default level, explicit debug, invalid level, valid LogDir, non-existent LogDir, LogDir is a file). - TestDdb_closePoolIfOpen: Close not called when already closed, called when open, Close error tolerated. Extract runDdb(ctx, args) as a production function containing the core execution logic so that tests exercise the real code path without triggering os.Exit. main() retains only OS-level setup (SetTraceback, DisableCStdoutBuffering). Add AssertStringContains(t, s string, substrings ...string) to common/test/utils.go for reuse across the control-plane test suite. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>
Before this change, rebuild_kill_PS_leader_during_rebuild() killed a non-leader engine and immediately (tried to) inject fault DAOS_REBUILD_TGT_SCAN_HANG on "all engines". This fault injection itself suffered RPC timeouts due to the killed engine. This further affected the test's overall timing, contradicting the goal of killing the PS leader engine during the first rebuild. With this change, the test no longer uses fault injection. Instead it waits for the first rebuild to start *and* demonstrate evidence of scanning activity (rs_toberb_obj_nr > 0). This is accomplished using a new common function test_rebuild_wait_to_scanning_next() that waits for both the rs_version (pool map version) to increment and the to-be-rebuilt number of objects to become nonzero. Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
We've observed the following events:
05:10:12.442962 swim_member_suspect() 6: member 0 2747111301814681600
is SUSPECT from 2
05:10:28.810595 ds_mgmt_group_update() updated group: 120 -> 122: 10
ranks
05:10:33.344560 swim_member_dead() 6: member 0 2747126810886799360 is
DEAD from 6 (self)
PG version 122 includes a rejoined member 0, whose incarnation changed
from 2747111301814681600 to 2747126810886799360. Since the suspicion
timeout was 20 s, how could there be a DEAD event for the newer
incarnation of member 0 within less than 5 s since the PG update? The
problem is that crt_swim_rank_check forgets to delete the existing
suspicion on member 0, as suggested by the first event, which happened
about one suspicion timeout before the DEAD event.
Fundamentally, such suspicions (and perhaps other swim_item objects)
should be about specific incarnations, not just member IDs. But that
might require a larger change. In this patch, we offer a quick fix,
leaving the larger change to a future patch.
Signed-off-by: Li Wei <liwei@hpe.com>
The test employs 30000 threads via concurrent.futures.ThreadPoolExecutor. Each thread creates a container and sleeps 2 s. In el8’s Python 3.6, this ThreadPoolExecutor allows (os.cpu_count() or 1) * 5 threads to run at the same time, whereas in el9’s Python 3.9, it only allows min(32, (os.cpu_count() or 1) + 4). Log messages produced by my debug PR confirm the latter to be 32, which must be way smaller than the former. If we consider only the time taken by the sleep in every thread, then it takes at least 2 * (30000 / 32) = 1875 s to finish 30000 such threads, already longer than the test timeout of 1200 s. This patch specifies max_workers explicitly to limit the delay caused by sleeping. Signed-off-by: Li Wei <liwei@hpe.com>
- The new "Cluster in a Box" stage is intended to be able to support testing that currently requires hardware resources to run. A Cluster in a Box is a KVM host that hosts VMs that have emulated NVMe and passthrough IB devices, so they appear more like HW systems. - Changed several "PR" tests to run on Cluster Box stages. Some of the tests' configurations were adjusted to be able to run on a VM with less resources. Some tests can take longer to run so timeouts were increased. - In the Jenkinsfile, added pragma to set build description Signed-off-by: Ryon Jensen <ryon.jensen@hpe.com>
Pin setuptools<82 since v82 remove pkg_resources, which we are still using. Signed-off-by: Dalton Bohning <dalton.bohning@hpe.com>
Add a /var/daos directory that is writeable for daos_server, which can then be used as a persistent location for control_meta and the mountpounts for the SCM/tmpfs filesystems (which should not be placed into /tmp). Signed-off-by: Michael Hennecke <michael.hennecke@hpe.com>
Run ldconfig on daos-client post install and uninstall. This fixes issues where other packages run ldconfig between daos-client installs and break the linkage. See DAOS-19289 for more details. Signed-off-by: Dalton Bohning <dalton.bohning@hpe.com>
Retern DER_INVAL when an invalid db path is provided to the ddb command. Signed-off-by: Jan Michalski <jan-marian.michalski@hpe.com>
For MD on SSD configurations, engine local SCM capacity based corpc timeout configuration: - Add pool_destroy_local_scm_size() get engine-local SCM size - Add pool_destroy_rpc_timeout() to customize destroy CoRPC timeout for ds_mgmt_tgt_pool_destroy_ranks() - non-MD-on-SSD case falls back to default timeout And serialize pool target destroy handling for when a previous handler invocation (whose CoRPC initiator timed out) is still busy performing expensive subtree destruction / file unlinking: - Refactor ds_pooltgts with dual synchronization domains: Create: dpt_create_mutex/cv + dpt_creates_ht (existing create-cancel) Destroy: dpt_destroy_mutex/cv + dpt_destroys_ht (new serialization) - Add ds_pooltgts_destroy_rec for destroy-in-flight tracking hash table - Add to ds_mgmt_hdlr_tgt_destroy() the destroy serialization Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
The dfuse progress thread currently polls DAOS EQ with NOWAIT while there are outstanding events. If the server is down or events make no progress, this can turn into busy polling. Track consecutive empty EQ polls and switch to a small bounded poll timeout after repeated empty polls. The timeout backs off from 50us and is capped at 5ms, while active completion traffic resets the counter and keeps the existing NOWAIT behavior. This reduces CPU usage during stalled/no-progress periods without changing the normal async submit path. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
Enable the ftest/control config generage tests to run on VMD enabled clusters in CI. Make use of the new --allow-numa-imbalance config generate option to run on the existing CI cluster with VMD. Test-tag: ConfigGenerateRun ConfigGenerateOutput Skip-functional-hardware-medium-vmd: false Skip-functional-hardware-medium: false Quick-functional: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
|
Ticket title is '2.8.0rc1 : control/config generate tests on VMD enabled cluster' |
Test-tag: ConfigGenerateRun ConfigGenerateOutput Skip-functional-hardware-medium-vmd: false Skip-functional-hardware-medium: false skip-Functional-on-EL-9: true skip-Functional-Hardware-Large-MD-on-SSD: true skip-Functional-Hardware-Medium-Verbs-Provider-MD-on-SSD: true Quick-functional: true
Collaborator
|
Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18647/3/execution/node/904/log |
Collaborator
|
Test stage Functional Hardware Medium VMD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18647/3/execution/node/893/log |
Collaborator
|
Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18647/3/execution/node/1083/log |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable the ftest/control config generage tests to run on VMD enabled clusters in CI. Make use of the new --allow-numa-imbalance config generate option to run on the existing CI cluster with VMD.
Test-tag: ConfigGenerateRun ConfigGenerateOutput
Skip-functional-hardware-medium-vmd: false
Skip-functional-hardware-medium: false
Quick-functional: true
Steps for the author:
After all prior steps are complete: