Skip to content

CMP-3866: Enhanced TestScanTailoredProfileIsDeprecated to cover case when TailoredProfile extends a Profile that is marked deprecated#1089

Open
taimurhafeez wants to merge 4 commits into
ComplianceAsCode:masterfrom
taimurhafeez:CMP-3866
Open

CMP-3866: Enhanced TestScanTailoredProfileIsDeprecated to cover case when TailoredProfile extends a Profile that is marked deprecated#1089
taimurhafeez wants to merge 4 commits into
ComplianceAsCode:masterfrom
taimurhafeez:CMP-3866

Conversation

@taimurhafeez
Copy link
Copy Markdown
Collaborator

@taimurhafeez taimurhafeez commented Feb 18, 2026

Test Coverage
Adds two test cases having distinct deprecation scenarios:

Scenario 1 :

  • TailoredProfile itself is marked deprecated via annotation
  • Verifies warning: "TailoredProfile X is deprecated and should be avoided."

Scenario 2 :

  • TailoredProfile extends a Profile that is marked deprecated
  • TailoredProfile itself is NOT marked deprecated
  • Verifies warning: "TailoredProfile X is extending a deprecated Profile (Y) that will be removed in a future version."

Run the test

If you encounter an error for running the test that relates to invalid machine selector etc, following worked for me.

In a separate terminal 1,
while true; do oc patch validatingadmissionpolicy custom-machine-config-pool-selector --type=merge -p '{"spec":{"failurePolicy":"Ignore"}}' 2>/dev/null; sleep 3; done &

Then in main terminal 2
make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestScanTailoredProfileIsDeprecated"
Or
make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestScanTailoredProfileExtendsDeprecatedProfile"
After the test run,
kill the process in terminal 1 by kill %1
Expected output:

=== RUN   TestScanTailoredProfileExtendsDeprecatedProfile
=== PAUSE TestScanTailoredProfileExtendsDeprecatedProfile
=== CONT  TestScanTailoredProfileExtendsDeprecatedProfile
2026/05/20 17:47:55 Creating ProfileBundle test-scan-tailored-profile-extends-deprecated-profile
2026/05/20 17:48:01 waiting ProfileBundle test-scan-tailored-profile-extends-deprecated-profile to become VALID (PENDING)
2026/05/20 17:48:06 waiting ProfileBundle test-scan-tailored-profile-extends-deprecated-profile to become VALID (PENDING)
2026/05/20 17:48:11 waiting ProfileBundle test-scan-tailored-profile-extends-deprecated-profile to become VALID (PENDING)
2026/05/20 17:48:16 ProfileBundle ready (VALID)
2026/05/20 17:48:22 TailoredProfile ready (READY)
    common.go:510: Found ComplianceScan deprecated profile event: TailoredProfile test-scan-tailored-profile-extends-deprecated-profile-tp is extending a deprecated Profile (test-scan-tailored-profile-extends-deprecated-profile-cis-1-4) that will be removed in a future version of Compliance Operator. Please consider using a newer version of this profile
2026/05/20 17:48:33 Waiting for run of test-scan-tailored-profile-extends-deprecated-profile-tp compliancescan (RUNNING)
2026/05/20 17:48:38 Waiting for run of test-scan-tailored-profile-extends-deprecated-profile-tp compliancescan (RUNNING)
2026/05/20 17:48:43 Waiting for run of test-scan-tailored-profile-extends-deprecated-profile-tp compliancescan (RUNNING)
2026/05/20 17:48:48 Waiting for run of test-scan-tailored-profile-extends-deprecated-profile-tp compliancescan (RUNNING)
2026/05/20 17:48:53 Waiting for run of test-scan-tailored-profile-extends-deprecated-profile-tp compliancescan (RUNNING)
2026/05/20 17:48:58 Waiting for run of test-scan-tailored-profile-extends-deprecated-profile-tp compliancescan (AGGREGATING)
2026/05/20 17:49:03 Waiting for run of test-scan-tailored-profile-extends-deprecated-profile-tp compliancescan (AGGREGATING)
2026/05/20 17:49:08 ComplianceScan ready (DONE)
--- PASS: TestScanTailoredProfileExtendsDeprecatedProfile (73.51s)
PASS

=== RUN   TestScanTailoredProfileIsDeprecated
=== PAUSE TestScanTailoredProfileIsDeprecated
=== CONT  TestScanTailoredProfileIsDeprecated
    common.go:540: Found ComplianceScan deprecated profile event: TailoredProfile test-tailored-profile-is-deprecated is deprecated and should be avoided. Please consider using another profile
2026/05/21 17:30:39 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/05/21 17:30:44 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/05/21 17:30:49 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/05/21 17:30:54 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/05/21 17:30:59 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/05/21 17:31:04 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/05/21 17:31:09 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/05/21 17:31:14 ComplianceScan ready (DONE)
--- PASS: TestScanTailoredProfileIsDeprecated (47.16s)
PASS


Assisted by Claude Code.

…oredProfile extends a Profile that is marked deprecated
@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Feb 18, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: taimurhafeez

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Feb 18, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@taimurhafeez taimurhafeez marked this pull request as ready for review February 18, 2026 11:39
@github-actions
Copy link
Copy Markdown

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1089-2436da3e23232cbad7a5a6d3d719c09d8494c6a2

@taimurhafeez taimurhafeez changed the title Enhanced TestScanTailoredProfileIsDeprecated to cover case when TailoredProfile extends a Profile that is marked deprecated CMP-3866: Enhanced TestScanTailoredProfileIsDeprecated to cover case when TailoredProfile extends a Profile that is marked deprecated Feb 18, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

@taimurhafeez: This pull request references CMP-3866 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@github-actions
Copy link
Copy Markdown

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1089-b9ff8fc1f6405ee594fdc887e352369ad6b12c38

@taimurhafeez
Copy link
Copy Markdown
Collaborator Author

/retest-required

@taimurhafeez
Copy link
Copy Markdown
Collaborator Author

/retest

Copy link
Copy Markdown
Collaborator

@rhmdnd rhmdnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments inline, mainly around the test images and the possibility of making this it's own test.

Thanks!

Comment thread tests/e2e/parallel/main_test.go Outdated
}

// Find one of the profiles created by the ProfileBundle and annotate it as deprecated
// Using ocp4-cis as it should be available in the standard content
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could list profiles in the bundle and grab one if we want to make this a little more resilient to upstream profile changes.

Comment thread tests/e2e/parallel/main_test.go Outdated
t.Fatal(err)
}

// SCENARIO 2: TailoredProfile extends a deprecated Profile - the TailoredProfile itself is not marked deprecated,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this being an entirely separate scenario - should it just be it's own test? In that case the profile bundle name could be derived from the test name.

Comment thread tests/e2e/parallel/main_test.go Outdated
}

// Create a TailoredProfile that extends the deprecated Profile
tpExtendsDeprecatedName := "test-tp-extends-deprecated"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar naming comment here.

Comment thread tests/e2e/parallel/main_test.go Outdated
if deprecatedProfileCopy.Annotations == nil {
deprecatedProfileCopy.Annotations = make(map[string]string)
}
deprecatedProfileCopy.Annotations[compv1alpha1.ProfileStatusAnnotation] = "deprecated"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use the deprecated_profile image?

https://github.com/ComplianceAsCode/compliance-operator/tree/master/images/testcontent/deprecated_profile

I think @yuumasato built an image with a data stream that contains a deprecated profile specifically for testing when he implemented the deprecated profile functionality.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

@taimurhafeez: This pull request references CMP-3866 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Test Coverage
The test now covers two distinct deprecation scenarios:

Scenario 1 (existing):

  • TailoredProfile itself is marked deprecated via annotation
  • Verifies warning: "TailoredProfile X is deprecated and should be avoided."

Scenario 2 (Added) :

  • TailoredProfile extends a Profile that is marked deprecated
  • TailoredProfile itself is NOT marked deprecated
  • Verifies warning: "TailoredProfile X is extending a deprecated Profile (Y) that will be removed in a future version."

Run the test
make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestScanTailoredProfileIsDeprecated"

Expected output:

=== RUN   TestScanTailoredProfileIsDeprecated
=== PAUSE TestScanTailoredProfileIsDeprecated
=== CONT  TestScanTailoredProfileIsDeprecated
   common.go:521: Found ComplianceScan deprecated profile event: TailoredProfile test-tailored-profile-is-deprecated is deprecated and should be avoided. Please consider using another profile
2026/02/18 11:27:18 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/02/18 11:27:23 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/02/18 11:27:28 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/02/18 11:27:33 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/02/18 11:27:38 Waiting for run of test-tailored-profile-is-deprecated compliancescan (RUNNING)
2026/02/18 11:27:43 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/02/18 11:27:48 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/02/18 11:27:53 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/02/18 11:27:58 Waiting for run of test-tailored-profile-is-deprecated compliancescan (AGGREGATING)
2026/02/18 11:28:03 ComplianceScan ready (DONE)
2026/02/18 11:28:08 waiting ProfileBundle test-deprecated-pb to become VALID (PENDING)
2026/02/18 11:28:13 waiting ProfileBundle test-deprecated-pb to become VALID (PENDING)
2026/02/18 11:28:18 waiting ProfileBundle test-deprecated-pb to become VALID (PENDING)
2026/02/18 11:28:23 waiting ProfileBundle test-deprecated-pb to become VALID (PENDING)
2026/02/18 11:28:28 waiting ProfileBundle test-deprecated-pb to become VALID (PENDING)
2026/02/18 11:28:33 waiting ProfileBundle test-deprecated-pb to become VALID (PENDING)
2026/02/18 11:28:38 waiting ProfileBundle test-deprecated-pb to become VALID (PENDING)
2026/02/18 11:28:43 ProfileBundle ready (VALID)
2026/02/18 11:28:48 TailoredProfile ready (READY)
   common.go:521: Found ComplianceScan deprecated profile event: TailoredProfile test-tp-extends-deprecated is extending a deprecated Profile (test-deprecated-pb-cis) that will be removed in a future version of Compliance Operator. Please consider using a newer version of this profile
2026/02/18 11:28:59 Waiting for run of test-tp-extends-deprecated compliancescan (RUNNING)
2026/02/18 11:29:04 Waiting for run of test-tp-extends-deprecated compliancescan (RUNNING)
2026/02/18 11:29:09 Waiting for run of test-tp-extends-deprecated compliancescan (RUNNING)
2026/02/18 11:29:14 Waiting for run of test-tp-extends-deprecated compliancescan (RUNNING)
2026/02/18 11:29:19 Waiting for run of test-tp-extends-deprecated compliancescan (AGGREGATING)
2026/02/18 11:29:24 Waiting for run of test-tp-extends-deprecated compliancescan (AGGREGATING)
2026/02/18 11:29:29 Waiting for run of test-tp-extends-deprecated compliancescan (AGGREGATING)
2026/02/18 11:29:34 Waiting for run of test-tp-extends-deprecated compliancescan (AGGREGATING)
2026/02/18 11:29:39 ComplianceScan ready (DONE)
--- PASS: TestScanTailoredProfileIsDeprecated (152.11s)
PASS

Assisted by Claude Code.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@github-actions
Copy link
Copy Markdown

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1089-f47951b9a9a984ad386721adf739eb92fef8aaa5

@taimurhafeez taimurhafeez requested a review from rhmdnd May 20, 2026 13:10
@taimurhafeez
Copy link
Copy Markdown
Collaborator Author

A few comments inline, mainly around the test images and the possibility of making this it's own test.

Thanks!

@rhmdnd Please review if modifications are acceptable.

@github-actions
Copy link
Copy Markdown

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1089-4bdcb9e2c66e82b36e268f2119b9288b0886c4c3

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented May 21, 2026

@taimurhafeez: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-rosa 4bdcb9e link true /test e2e-rosa
ci/prow/e2e-aws-parallel-arm 4bdcb9e link true /test e2e-aws-parallel-arm
ci/prow/e2e-aws-parallel 4bdcb9e link true /test e2e-aws-parallel
ci/prow/e2e-aws-serial-arm 4bdcb9e link true /test e2e-aws-serial-arm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants