CM-261: utilize ginkgo label filter to differentiate cloud platforms

[lunarwhite]

Main changes:

• Bump onsi/gingko/v2 to v2.19.
  • To make use of the newly introduced Label Sets feature. Benefits:
    • It is more expressive and human-readable than regexp combining --focus and --skip.
    • It extends the current --label-filter to support filtering by key-value pairs, with the contains and not contains operations for the labels that have values. (more example usages can be found in the doc)
    • It gives more flexibility in filtering and sorting tests based on labels. As the test suite grows, we might apply more label types to identify Serial/Disruptive, Alpha/Beta, Conformance/MinVersion:xxx, etc.
• Add an env variable E2E_GINKGO_LABEL_FILTER in the Makefile. Append -ginkgo.label-filter=$(E2E_GINKGO_LABEL_FILTER) to the go test command.
• Apply labels "Platform:AWS", "Platform:GCP" and "Platform:IBM" to cases description accordingly.
• Set the default value of E2E_GINKGO_LABEL_FILTER in the Makefile to "Platform: isSubsetOf {AWS}".
  • It would then select and run both non-platform-specific and AWS-only cases.

Other changes:

• Remove some hardcoded skipping logic.
• Regroup ACME dns-01 cases by their provider platform. Each dns-01 provider now will have a dedicated Context().

I've tested on the GCP cluster. It shows: (skipped 3 cases labeled "Platform:AWS", 1 labeled "Platform:IBM")

go test \
-timeout 1h \
-count 1 \
-v \
-p 1 \
-tags e2e \
-run "" \
./test/e2e \
-ginkgo.label-filter="Platform: isSubsetOf {GCP}"

...

Ran 17 of 21 Specs in 524.481 seconds
SUCCESS! -- 17 Passed | 0 Failed | 0 Pending | 4 Skipped
--- PASS: TestAll (524.49s)
PASS
ok  	github.com/openshift/cert-manager-operator/test/e2e	846.604s

[openshift-ci-robot]

@lunarwhite: This pull request references CM-261 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

Details

In response to this:

Main changes:

• Bump onsi/gingko/v2 to v2.19.
• To make use of the newly introduced Label Sets feature. Benefits:
  • It is more expressive and human-readable than regexp combining --focus and --skip.
  • It extends the current --label-filter to support filtering by key-value pairs, with the contains and not contains operations for the labels that have values. (more example usages can be found in the doc)
  • It gives more flexibility in filtering and sorting tests based on labels. As the test suite grows, we might apply more label types to identify Serial/Disruptive, Alpha/Beta, Conformance/MinVersion:xxx, etc.
• Add an env variable E2E_GINKGO_LABEL_FILTER in the Makefile. Append -ginkgo.label-filter=$(E2E_GINKGO_LABEL_FILTER) to the go test command.
• Apply labels "Platform:AWS", "Platform:GCP" and "Platform:IBM" to cases description accordingly.
• Set the default value of E2E_GINKGO_LABEL_FILTER in the Makefile to "Platform: isSubsetOf {AWS}".
• It would then select and run both non-platform-specific and AWS-only cases.

Other changes:

• Remove some hardcoded skipping logic.
• Regroup ACME dns-01 cases by their provider platform. Each dns-01 provider now will have a dedicated Context().

I've tested on the GCP cluster. It shows: (skipped 3 cases labeled "Platform:AWS", 1 labeled "Platform:IBM")

go test \
-timeout 1h \
-count 1 \
-v \
-p 1 \
-tags e2e \
-run "" \
./test/e2e \
-ginkgo.label-filter="Platform: isSubsetOf {GCP}"

...

Ran 17 of 21 Specs in 524.481 seconds
SUCCESS! -- 17 Passed | 0 Failed | 0 Pending | 4 Skipped
--- PASS: TestAll (524.49s)
PASS
ok  	github.com/openshift/cert-manager-operator/test/e2e	846.604s

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[lunarwhite]

/uncc deads2k
/cc @swghosh @TrilokGeer

CI is green. Mark it as ready for review now.

To keep it simple and easy for review and updates, I created this separate PR for CM-261. I didn't touch the main logic of the existing cases, so your comments in #181 are not fully addressed. After this is done, we can rebase and update #181.

[swghosh]

Generally, LGTM
great work @lunarwhite thanks for this!
I'll take another closer look into the test/e2e/certificates_test.go later during the week.

[swghosh]

Does addition of this skip the non-ginkgo tests present in test/e2e/cert_manager_deployment_test.go?
It seems fine to me if it does, because those tests are essentially repetition of what's also in the ginkgo suite so we can skip them during e2e runs moving forward (we have had this behaviour tested for long and it isn't a problem).

[lunarwhite]

This filter only applies to tests in the Ginkgo framework. The test/e2e/cert_manager_deployment_test.go go tests would not be affected.

[swghosh]

Not sure why these whitespace changes? (maybe your local linter is mis-behaving)

[lunarwhite]

Will undo these changes as they are not related to this PR. (Yeah. I've set the "files.trimTrailingWhitespace": true in my vscode editor)

[swghosh]

/test all

[swghosh]

/lgtm
/label docs-approved
/label px-approved
(non-user facing, only e2e test related changes)

[swghosh]

/assign @TrilokGeer

[swghosh]

/approve
/hold
/test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lunarwhite, swghosh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [swghosh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[swghosh]

/remove-hold
/retest-required

[lunarwhite]

/retest-required

[lunarwhite]

self-adding
/label qe-approved

[openshift-ci-robot]

@lunarwhite: This pull request references CM-261 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

Details

In response to this:

Main changes:

• Bump onsi/gingko/v2 to v2.19.
• To make use of the newly introduced Label Sets feature. Benefits:
  • It is more expressive and human-readable than regexp combining --focus and --skip.
  • It extends the current --label-filter to support filtering by key-value pairs, with the contains and not contains operations for the labels that have values. (more example usages can be found in the doc)
  • It gives more flexibility in filtering and sorting tests based on labels. As the test suite grows, we might apply more label types to identify Serial/Disruptive, Alpha/Beta, Conformance/MinVersion:xxx, etc.
• Add an env variable E2E_GINKGO_LABEL_FILTER in the Makefile. Append -ginkgo.label-filter=$(E2E_GINKGO_LABEL_FILTER) to the go test command.
• Apply labels "Platform:AWS", "Platform:GCP" and "Platform:IBM" to cases description accordingly.
• Set the default value of E2E_GINKGO_LABEL_FILTER in the Makefile to "Platform: isSubsetOf {AWS}".
• It would then select and run both non-platform-specific and AWS-only cases.

Other changes:

• Remove some hardcoded skipping logic.
• Regroup ACME dns-01 cases by their provider platform. Each dns-01 provider now will have a dedicated Context().

I've tested on the GCP cluster. It shows: (skipped 3 cases labeled "Platform:AWS", 1 labeled "Platform:IBM")

go test \
-timeout 1h \
-count 1 \
-v \
-p 1 \
-tags e2e \
-run "" \
./test/e2e \
-ginkgo.label-filter="Platform: isSubsetOf {GCP}"

...

Ran 17 of 21 Specs in 524.481 seconds
SUCCESS! -- 17 Passed | 0 Failed | 0 Pending | 4 Skipped
--- PASS: TestAll (524.49s)
PASS
ok  	github.com/openshift/cert-manager-operator/test/e2e	846.604s

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@lunarwhite: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[lunarwhite]

/cherry-pick cert-manager-1.15

[openshift-cherrypick-robot]

@lunarwhite: new pull request created: #251

Details

In response to this:

/cherry-pick cert-manager-1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.