Skip to content

[CI][benchmarks] Modify benchmark scripts and CI workflow to enable gdb_mode #20891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
luszczewskakasia1 wants to merge 1 commit into
base: sycl
Choose a base branch
from
+77 −13
Open

[CI][benchmarks] Modify benchmark scripts and CI workflow to enable gdb_mode #20891

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/sycl-linux-run-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ on:
Extra options to be added to LIT_OPTS.
type: string
default: ''
gdb_mode:
type: string
required: False

repo_ref:
type: string
Expand Down Expand Up @@ -134,6 +137,10 @@ on:
type: string
default: 'false'
required: False
benchmark_gdb_mode:
type: string
required: False
default: 'false'

in_workflow_call_mode:
description: |
Expand Down Expand Up @@ -379,3 +386,4 @@ jobs:
exit_on_failure: ${{ inputs.benchmark_exit_on_failure }}
build_ref: ${{ inputs.repo_ref }}
runner: ${{ inputs.runner }}
gdb_mode: ${{ inputs.benchmark_gdb_mode }}
11 changes: 10 additions & 1 deletion .github/workflows/sycl-ur-perf-benchmarking.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,12 @@ on:
- 'false'
- 'true'
default: 'false'
gdb_mode:
type: choice
options:
- false
- true
default: false

concurrency:
# Cancel a currently running workflow for:
Expand Down Expand Up @@ -183,7 +189,7 @@ jobs:
uses: ./.github/workflows/sycl-linux-run-tests.yml
secrets: inherit
with:
name: "Benchmarks (${{ matrix.runner }}, ${{ matrix.backend }}, preset: ${{ inputs.preset }})"
name: "Benchmarks (${{ matrix.runner }}, ${{ matrix.backend }}, preset: ${{ inputs.preset }}, gdb_mode: ${{ inputs.gdb_mode }})"
runner: ${{ matrix.runner }}
image: ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest
image_options: -u 1001 --device=/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --privileged --cap-add SYS_ADMIN
Expand All @@ -193,6 +199,7 @@ jobs:
benchmark_save_name: ${{ needs.sanitize_inputs_dispatch.outputs.benchmark_save_name }}
benchmark_preset: ${{ inputs.preset }}
benchmark_exit_on_failure: ${{ inputs.exit_on_failure }}
benchmark_gdb_mode: ${{ inputs.gdb_mode }}
repo_ref: ${{ needs.sanitize_inputs_dispatch.outputs.build_ref }}
toolchain_artifact: ${{ needs.build_sycl_dispatch.outputs.toolchain_artifact }}
toolchain_artifact_filename: ${{ needs.build_sycl_dispatch.outputs.toolchain_artifact_filename }}
Expand Down Expand Up @@ -241,6 +248,7 @@ jobs:
benchmark_upload_results: true
benchmark_save_name: ${{ matrix.save_name }}
benchmark_preset: ${{ matrix.preset }}
benchmark_gdb_mode: false
repo_ref: ${{ matrix.ref }}
toolchain_artifact: ${{ needs.build_nightly.outputs.toolchain_artifact }}
toolchain_artifact_filename: ${{ needs.build_nightly.outputs.toolchain_artifact_filename }}
Expand All @@ -267,5 +275,6 @@ jobs:
benchmark_preset: 'Minimal'
benchmark_dry_run: true
benchmark_exit_on_failure: true
benchmark_gdb_mode: ${{ inputs.gdb_mode }}
repo_ref: ${{ github.sha }}
# END benchmark framework builds and runs on PRs path
31 changes: 20 additions & 11 deletions devops/actions/run-tests/benchmark/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ inputs:
type: string
required: False
default: ""
gdb_mode:
type: string
required: False
# dry-run is passed only to compare.py (to not fail on regression), not to main.py (where such flag would omit all benchmark runs)
dry_run:
type: string
Expand Down Expand Up @@ -267,6 +270,7 @@ runs:
env:
BENCH_WORKDIR: ${{ steps.establish_outputs.outputs.BENCH_WORKDIR }}
BENCHMARK_RESULTS_REPO_PATH: ${{ steps.establish_outputs.outputs.BENCHMARK_RESULTS_REPO_PATH }}
LLVM_BENCHMARKS_USE_GDB: ${{ inputs.gdb_mode }}
run: |
# Build and run benchmarks

Expand All @@ -280,6 +284,7 @@ runs:
echo "::endgroup::"
echo "::group::run_benchmarks"

export LLVM_BENCHMARKS_USE_GDB=${LLVM_BENCHMARKS_USE_GDB}
Copy link
Contributor

@lukaszstolarczuk lukaszstolarczuk Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when is LLVM_BENCHMARKS_USE_GDB set?

Copy link
Contributor Author

@luszczewskakasia1 luszczewskakasia1 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must have deleted it accidentally. Now it is added

numactl --cpunodebind "$NUMA_NODE" --membind "$NUMA_NODE" \
./devops/scripts/benchmarks/main.py "$BENCH_WORKDIR" \
--sycl "$(realpath $CMPLR_ROOT)" \
Expand All @@ -299,17 +304,21 @@ runs:
echo "::endgroup::"
echo "::group::compare_results"

python3 ./devops/scripts/benchmarks/compare.py to_hist \
--avg-type EWMA \
--cutoff "$(date -u -d '7 days ago' +'%Y%m%d_%H%M%S')" \
--name "$SAVE_NAME" \
--compare-file "${BENCHMARK_RESULTS_REPO_PATH}/results/${SAVE_NAME}_${SAVE_TIMESTAMP}.json" \
--results-dir "${BENCHMARK_RESULTS_REPO_PATH}/results/" \
--regression-filter '^[a-z_]+_sycl .* CPU count' \
--regression-filter-type 'SYCL benchmark (measured using CPU cycle count)' \
--verbose \
--produce-github-summary \
${{ inputs.dry_run == 'true' && '--dry-run' || '' }} \
if [ "$LLVM_BENCHMARKS_USE_GDB" == "false" ]; then
python3 ./devops/scripts/benchmarks/compare.py to_hist \
--avg-type EWMA \
--cutoff "$(date -u -d '7 days ago' +'%Y%m%d_%H%M%S')" \
--name "$SAVE_NAME" \
--compare-file "${BENCHMARK_RESULTS_REPO_PATH}/results/${SAVE_NAME}_${SAVE_TIMESTAMP}.json" \
--results-dir "${BENCHMARK_RESULTS_REPO_PATH}/results/" \
--regression-filter '^[a-z_]+_sycl .* CPU count' \
--regression-filter-type 'SYCL benchmark (measured using CPU cycle count)' \
--verbose \
--produce-github-summary \
${{ inputs.dry_run == 'true' && '--dry-run' || '' }}
else
echo "Skipping regression comparison due to GDB mode enabled."
fi

echo "::endgroup::"
- name: Run benchmarks integration tests
Expand Down
15 changes: 15 additions & 0 deletions devops/scripts/benchmarks/benches/compute.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import csv
import io
import math
import os
from enum import Enum
from itertools import product
from pathlib import Path
Expand Down Expand Up @@ -76,6 +77,7 @@ def setup(self) -> None:
"compute-benchmarks",
use_installdir=False,
)
print(self._project)
Copy link
Contributor

@PatKamin PatKamin Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these some debug prints? Why logger is not used?


if not self._project.needs_rebuild():
log.info(f"Rebuilding {self._project.name} skipped")
Expand All @@ -89,6 +91,13 @@ def setup(self) -> None:
f"-DCMAKE_CXX_COMPILER=clang++",
f"-DCMAKE_C_COMPILER=clang",
]

is_gdb_mode = os.environ.get("LLVM_BENCHMARKS_USE_GDB", "") == "true"
if is_gdb_mode:
extra_args += [
f"-DCMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG -fdebug-info-for-profiling",
]

if options.ur_adapter == "cuda":
extra_args += [
"-DBUILD_SYCL_WITH_CUDA=ON",
Expand Down Expand Up @@ -567,6 +576,12 @@ def __enabled_runtimes(self) -> list[RUNTIMES]:
return runtimes

def __parse_output(self, output: str) -> list[tuple[float, float]]:
is_gdb_mode = os.environ.get("LLVM_BENCHMARKS_USE_GDB", "") == "true"
print(output)

if is_gdb_mode:
return [(0.0, 0.0)]

csv_file = io.StringIO(output)
reader = csv.reader(csv_file)
next(reader, None)
Expand Down
6 changes: 5 additions & 1 deletion devops/scripts/benchmarks/git_project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,15 @@ def configure(
add_sycl: bool = False,
) -> None:
"""Configures the project."""

is_gdb_mode = os.environ.get("LLVM_BENCHMARKS_USE_GDB", "") == "true"
build_type = "RelWithDebInfo" if is_gdb_mode else "Release"

cmd = [
"cmake",
f"-S {self.src_dir}",
f"-B {self.build_dir}",
f"-DCMAKE_BUILD_TYPE=Release",
f"-DCMAKE_BUILD_TYPE={build_type}",
]
if self._use_installdir:
cmd.append(f"-DCMAKE_INSTALL_PREFIX={self.install_dir}")
Expand Down
19 changes: 19 additions & 0 deletions devops/scripts/benchmarks/utils/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,25 @@ def run(
if isinstance(command, str):
command = command.split()

is_gdb_mode = os.environ.get("LLVM_BENCHMARKS_USE_GDB", "") == "true"
if command[0] == "taskset" and is_gdb_mode:
Copy link
Contributor

@PatKamin PatKamin Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why taskset is crucial here? We shouldn't rely on taskset being used in benchmark commands (if this is the case), this could be changed in future.

If needed, perhaps add some wrapper function that modifies benchmarks command (or passes it through down here).

Copy link
Contributor

@lukaszstolarczuk lukaszstolarczuk Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand this check is to distinguish if we run a benchmark or some other command. I think we can (instead of looking for taskset) just check if a binary from .../compute-benchmarks-build/bin/ is run.

gdb_cmd = [
"gdb",
"-return-child-result",
"--batch",
"--ex",
"set confirm off",
"--ex",
"run",
"--ex",
"bt",
"--ex",
"quit",
"--args",
] + command
command = gdb_cmd
print(f"Running in gdb mode: {command}")

env = os.environ.copy()
for ldlib in ld_library:
if os.path.isdir(ldlib):
Expand Down
Loading