feat: E2E coverage instrumentation with CI pipeline for all dynamic plugins

[gustavolira]

Summary

Adds Istanbul-based E2E coverage collection for dynamic plugins with a CI pipeline that automatically builds instrumented OCI images when source.json changes — and skips builds when the image already exists.

Scope: frontend plugins only. Coverage collection uses window.__coverage__ which is only available for browser-executed code. Backend plugin coverage would require a different collection mechanism (global.__coverage__ from the Node.js process) and is out of scope for this PR.

How it works

1. source.json changes (push to main) trigger the build-instrumented-plugins workflow
2. For each workspace with e2e-tests/, the workflow checks if an instrumented OCI image already exists for the current repo-ref
3. If not, it runs instrument-plugin.sh which: clones upstream at the exact ref, builds the plugin (backstage-cli + janus-cli export-dynamic), then applies nyc instrument on the final webpack output
4. The instrumented bundle is published as an OCI artifact to ghcr.io
5. During E2E tests with E2E_COLLECT_COVERAGE=1, the _coverageCollector auto-fixture (in e2e-test-utils PR #95) collects window.__coverage__ from the browser after each test
6. report-coverage.sh merges per-test coverage JSONs via nyc merge, generates lcov via nyc report
7. upload-coverage.sh uploads lcov to Codecov with per-workspace flags (e2e-tech-radar, e2e-topology, etc.) for dashboard filtering

Key design decisions

• Post-webpack instrumentation: babel-plugin-istanbul gets stripped by webpack's module federation during export-dynamic-plugin. Applying nyc instrument after the final webpack build preserves Istanbul __coverage__ in the browser runtime
• Caching by source.json ref: OCI images are tagged with ref-<sha-first-12>. If the ref hasn't changed, the build is skipped entirely — no CI overhead
• Cross-repo Codecov attribution: Coverage uploads use the upstream repo's SHA and slug from source.json, so coverage appears on the correct repo in Codecov
• Parallel matrix: All 18 workspaces with E2E tests build in parallel via GitHub Actions matrix strategy
• nyc CLI for merge/report: Coverage merging and lcov generation use nyc merge + nyc report instead of custom code — nyc is already a pipeline dependency and handles lcov edge cases better
• testInfo.project.outputDir for coverage path: The auto-fixture writes coverage JSONs to <outputDir>/coverage/, avoiding CWD mismatch issues between Playwright workers and the main process

Known limitations

• Overlays and patches are not applied: The instrumented build clones upstream source directly. Production builds apply overlays from workspaces/*/plugins/*/overlay/ and patches from workspaces/*/patches/*.patch. Most workspaces have no overlays/patches, so the instrumented build matches production in the majority of cases. Tracking as a follow-up.
• Single frontend plugin per workspace: The build discovers the first frontend-plugin in metadata. Workspaces with multiple frontend plugins only instrument the first one.
• Missing injection integration: This PR builds instrumented OCI artifacts and provides the collection/upload mechanism, but the logic to swap production plugins with instrumented ones during E2E execution is a follow-up in e2e-test-utils.

Files

File	Purpose
.github/workflows/build-instrumented-plugins.yaml	CI workflow: detect changed workspaces, check OCI cache, build and publish
scripts/instrument-plugin.sh	Build instrumented dynamic plugin from upstream source
scripts/report-coverage.sh	Merge per-test coverage JSONs (nyc merge), generate lcov (nyc report), upload to Codecov
scripts/upload-coverage.sh	Upload lcov to Codecov with cross-repo attribution and flags

Related PRs

• rhdh-e2e-test-utils PR #95 — _coverageCollector auto-fixture that collects window.__coverage__ per test

Usage

# Manual: instrument a specific plugin
./scripts/instrument-plugin.sh tech-radar

# Run E2E with coverage collection
E2E_COLLECT_COVERAGE=1 ./run-e2e.sh -w tech-radar

# Merge + upload (called automatically by run-e2e.sh)
./scripts/report-coverage.sh tech-radar

# Upload only (requires lcov already generated)
CODECOV_TOKEN=xxx ./scripts/upload-coverage.sh tech-radar

Test plan

• Verify workflow detects changed source.json files correctly
• Verify OCI image existence check skips unnecessary builds
• Run ./scripts/instrument-plugin.sh tech-radar locally — verify __coverage__ in output
• Verify force-rebuild workflow_dispatch input bypasses cache
• Verify Codecov flags produce per-plugin dashboard filtering

Ref: RHIDP-13411

[gustavolira]

/review
-i
--pr_reviewer.require_score_review=true
--pr_reviewer.require_can_be_split_review=true
--pr_reviewer.num_code_suggestions="2"

[subhashkhileri]

Review:

the Playwright integration can be significantly simplified by replacing custom code with nyc CLI. See inline comments.

Also: Coverage collection (E2E_COLLECT_COVERAGE=1) should only be enabled on the e2e-ocp-helm PR check job — nightly uses released OCI refs (not instrumented builds), so coverage would produce empty data.

[subhashkhileri]

Review: Instrumentation Pipeline

Following up on the Playwright integration review — this covers the instrumentation workflow and build script.

The post-webpack instrumentation approach (nyc instrument after export-dynamic-plugin) is the right technique for module federation. However, there are significant gaps between how this workflow builds plugins vs. how the existing production workflow does it.

[subhashkhileri]

Proposal: Reuse the existing build pipeline instead of a separate instrumentation workflow

The current PR introduces scripts/instrument-plugin.sh (205 lines) and .github/workflows/build-instrumented-plugins.yaml (271 lines) — a parallel pipeline that re-clones, re-builds, and re-packages plugins from scratch. This duplicates the production build pipeline and misses overlays, patches, and edge cases that the production pipeline already handles.

Proposed approach

Instead of rebuilding from source, instrument the already-published production image. Add an instrument job to publish-release-branch-workspace-plugins.yaml that runs after the export job:

  instrument:
    needs: export
    if: # only for workspaces that have e2e-tests/
    runs-on: ubuntu-latest
    strategy:
      matrix:
        workspace: # detect workspaces with e2e-tests/ (similar to current detect-workspaces job)
      fail-fast: false
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v6

      - name: Resolve published image ref
        id: meta
        run: |
          # Read source.json + metadata to find the frontend plugin's image name and tag
          # Use the SAME naming/tagging as production: ghcr.io/{repo}/{plugin}:{tag}
          # The production image was just pushed by the export job

      - name: Log in to GHCR
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Setup Node.js
        uses: actions/setup-node@v6

      - name: Instrument and publish coverage image
        run: |
          PROD_IMAGE="${{ steps.meta.outputs.image-ref }}"
          COVERAGE_IMAGE="${{ steps.meta.outputs.coverage-image-ref }}"

          # 1. Extract plugin bundle from production image
          CONTAINER=$(podman create "$PROD_IMAGE")
          mkdir -p .instrumented
          podman cp "$CONTAINER:/opt/app-root/src/" .instrumented/plugin/
          podman rm "$CONTAINER"

          # 2. Instrument JS files with Istanbul
          npx --yes nyc instrument .instrumented/plugin/ .instrumented/instrumented/ --source-map

          # 3. Build new image from instrumented files
          cat > .instrumented/Containerfile <<EOF
          FROM scratch
          COPY instrumented/ /opt/app-root/src/
          EOF
          podman build -t "$COVERAGE_IMAGE" -f .instrumented/Containerfile .instrumented/

          # 4. Push with -coverage suffix, same tag as production
          podman push "$COVERAGE_IMAGE"

What this eliminates

File	Lines	Reason
scripts/instrument-plugin.sh	205	No need to re-clone, re-build, re-export from source
.github/workflows/build-instrumented-plugins.yaml	271	No separate workflow needed

What this fixes

1. Overlays and patches are included — the production image already has them applied
2. All plugins discovered — not just the first frontend-plugin match
3. Same toolset — Podman throughout, matching the production pipeline (no ORAS)
4. Same tag format — bs_1.49.4__1.32.0 with -coverage suffix on the image name
5. Same triggers — runs whenever production images are published (push to release branches)
6. No pnpm/npm detection needed — the plugin is already built

Coverage image naming

Production: ghcr.io/{repo}/{plugin}:{tag}
Coverage: ghcr.io/{repo}/{plugin}-coverage:{tag}

Same tag, -coverage suffix on the image name. This makes it trivial to swap in the coverage image during E2E runs — just append -coverage to the image name.

Note on image structure

The podman cp + podman build FROM scratch approach needs to match whatever directory layout the rhdh-cli package command produces. The exact path inside the container (e.g., /opt/app-root/src/) needs verification against the actual production image. The Containerfile above is illustrative — the actual paths should be confirmed by inspecting a published production image with podman inspect or skopeo inspect.

An alternative simpler approach: instead of rebuilding from a Containerfile, use podman commit after modifying the extracted files in-place, or use buildah to add/replace layers. The goal is to match the production image format exactly.

[subhashkhileri]

Move coverage logic out of run-e2e.sh

The ~30 lines of coverage code added to run-e2e.sh (nyc merge, nyc report, multi-workspace warning, per-workspace upload loop) should move into a self-contained script — e.g. rename scripts/upload-coverage.sh → scripts/report-coverage.sh and have it handle the full pipeline (merge → report → upload).

run-e2e.sh then reduces to:

if [[ "${E2E_COLLECT_COVERAGE:-}" == "1" ]]; then
    "$SCRIPT_DIR/scripts/report-coverage.sh" "${E2E_WORKSPACES[@]}"
fi

Keeps run-e2e.sh focused on test orchestration, makes the coverage script independently re-runnable (useful for debugging uploads without re-running tests), and avoids splitting the logic across two files.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[kadel]

This uses completely different way of packaging plugins than what production artifact uses.

The whole process of rebuilding plugins for testing seems looks problematic to me. I'm afraid that it will create a lot of problems down the line. It will be really easy to get those two ways of building out of sync (they already are) which will result in testing something different from what we ship.

We should consider alternative. I've tried to describe it in #2383 (comment)

[kadel]

This re-implements the full build pipeline that already exists in the production OCI image published by rhdh-plugin-export-utils. The only extra is nyc instrument.

What instead of reimplementing the build process and rebuilding we re-use existing artifacts.
We could pull the production image, extract it, add instrumentation and package it back to image with coverage. This way we don't rebuild the whole plugin. And minimize the difference between what is tested and what is shipped.

Something like this:

OCI_REF=ghcr.io/redhat-developer/rhdh-plugin-export-overlays/backstage-community-plugin-quay:next__1.28.1
COVERAGE_IMAGE_REF=ghcr.io/redhat-developer/rhdh-plugin-export-overlays/backstage-community-plugin-quay-coverage:next__1.28.1

# Pull production image
podman pull "$OCI_REF"

# Create a container (don't start it)
CID=$(podman create "$OCI_REF")

# Copy static/ out, instrument, copy back
podman cp "$CID:backstage-community-plugin-quay/dist/static" ./static-original/
npx nyc instrument ./static-original ./static-instrumented --source-map
podman cp ./static-instrumented/static "$CID:/backstage-community-plugin-quay/dist/"

# Commit as new image and push
podman commit "$CID" "$COVERAGE_IMAGE_REF"
podman rm "$CID"
# podman push "$COVERAGE_IMAGE_REF"