Skip to content

docs: add lab 10 topology-aware scheduling tutorial#610

Open
maishivamhoo123 wants to merge 5 commits into
Project-HAMi:masterfrom
maishivamhoo123:feat/lab3-topology-scheduling
Open

docs: add lab 10 topology-aware scheduling tutorial#610
maishivamhoo123 wants to merge 5 commits into
Project-HAMi:masterfrom
maishivamhoo123:feat/lab3-topology-scheduling

Conversation

@maishivamhoo123

@maishivamhoo123 maishivamhoo123 commented Jul 15, 2026

Copy link
Copy Markdown
Member

This tutorial provides a simple, step-by-step guide to testing HAMi’s smart GPU scheduling system on a regular computer without needing any real, expensive graphics cards. Inside, it contains instructions on how to simulate eight fake A100 GPUs and trick the system into thinking one specific GPU has a very poor network connection. It then shows you how to verify that the scheduler is smart enough to save the best-connected slots for heavy workloads while pushing single-GPU tasks onto the isolated one. This is incredibly useful because it allows developers and users to test and optimize their cluster setup completely for free, making it easy to catch bugs and automate tests before moving to real, expensive hardware.
Closes : #609

Summary by CodeRabbit

  • New Features
    • Added Lab 9: GPU Topology-Aware Scheduling on Fake GPUs.
    • Provides step-by-step instructions for configuring a local Kubernetes environment, simulating GPUs, and testing topology-aware scheduling.
    • Includes verification examples showing how GPU selection changes based on topology and cleanup instructions.
    • Added the lab to the Tutorials sidebar under “Labs,” labeled Intermediate with an estimated duration of about 45 minutes.

Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
@netlify

netlify Bot commented Jul 15, 2026

Copy link
Copy Markdown

Deploy Preview for project-hami ready!

Name Link
🔨 Latest commit cd41c7c
🔍 Latest deploy log https://app.netlify.com/projects/project-hami/deploys/6a572971ea7a8700093f1d67
😎 Deploy Preview https://deploy-preview-610--project-hami.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@hami-robot hami-robot Bot added the size/XL label Jul 15, 2026
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@maishivamhoo123, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 16 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 128e63b6-0344-4e66-892d-d506e4f04940

📥 Commits

Reviewing files that changed from the base of the PR and between 84699f9 and cd41c7c.

📒 Files selected for processing (2)
  • sidebars-tutorials.js
  • tutorials/labs/topology-aware-scheduling.md
📝 Walkthrough

Walkthrough

Adds Lab 9 to the tutorials sidebar and introduces a 615-line guide for configuring nvml-mock and HAMi to demonstrate topology-aware multi-GPU and single-GPU scheduling on simulated GPUs.

Changes

Topology-Aware Scheduling Lab

Layer / File(s) Summary
Lab registration and environment setup
sidebars-tutorials.js, tutorials/labs/topology-aware-scheduling.md
Registers the new Lab 9 document and documents metadata, prerequisites, local cluster creation, and the installation workflow.
Simulated GPU and HAMi installation
tutorials/labs/topology-aware-scheduling.md
Builds and installs nvml-mock, installs HAMi with topology-aware scheduling, and verifies simulated GPU capacity.
Custom topology and scheduler configuration
tutorials/labs/topology-aware-scheduling.md
Assigns low topology scores to GPU7, freezes device-plugin score updates, restarts scheduling, enables single-GPU topology awareness, and increases logging verbosity.
Scheduling verification and teardown
tutorials/labs/topology-aware-scheduling.md
Runs multi-GPU and single-GPU verification Pods, checks allocations and scheduler logs, and documents cleanup and next steps.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

Possibly related PRs

Suggested labels: lgtm, approved

Poem

I’m a bunny hopping through GPU terrain,
Making topology scores crystal clear again.
GPU7 hides, then hops in sight,
Two pods test the paths just right.
HAMi logs sparkle, clusters cheer—
A fluffy new lab is here!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed It identifies the main change as adding a topology-aware scheduling tutorial doc.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@hami-robot

hami-robot Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: maishivamhoo123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (3)
tutorials/labs/topology-aware-scheduling.md (2)

20-20: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Split imports into separate lines.

While technically valid in MDX 2, placing multiple import statements on the same line reduces readability. Consider splitting them.

🧹 Proposed format
-import Tabs from '`@theme/Tabs`'; import TabItem from '`@theme/TabItem`';
+import Tabs from '`@theme/Tabs`';
+import TabItem from '`@theme/TabItem`';
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tutorials/labs/topology-aware-scheduling.md` at line 20, Split the combined
imports in the MDX file into separate lines, keeping the existing Tabs and
TabItem imports unchanged and preserving their order.

485-486: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use kubectl wait instead of manual watch. Both steps rely on kubectl get pod -w, which blocks the terminal and requires a manual Ctrl+C interrupt (which isn't even mentioned in the single-GPU step). Replacing this with kubectl wait creates a better, automation-friendly user experience.

  • tutorials/labs/topology-aware-scheduling.md#L485-L486: Replace kubectl get pod multi-gpu -w with kubectl wait --for=condition=Ready pod/multi-gpu --timeout=60s.
  • tutorials/labs/topology-aware-scheduling.md#L534-L535: Replace kubectl get pod single-gpu -w with kubectl wait --for=condition=Ready pod/single-gpu --timeout=60s.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tutorials/labs/topology-aware-scheduling.md` around lines 485 - 486, Replace
the manual pod watch with kubectl wait using a 60-second timeout: update
tutorials/labs/topology-aware-scheduling.md lines 485-486 for pod/multi-gpu and
lines 534-535 for pod/single-gpu, preserving the subsequent annotation commands.
sidebars-tutorials.js (1)

54-58: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Fix indentation.

Align this block with the adjacent array items for consistent formatting (8 spaces instead of 10 for the opening brace).

🧹 Proposed format
-          {
-          type: "doc",
-          id: "labs/topology-aware-scheduling",
-          customProps: { level: "Intermediate", duration: "about 45 minutes" },
-        },
+        {
+          type: "doc",
+          id: "labs/topology-aware-scheduling",
+          customProps: { level: "Intermediate", duration: "about 45 minutes" },
+        },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@sidebars-tutorials.js` around lines 54 - 58, Adjust the indentation of the
sidebar item block beginning with the object containing id
"labs/topology-aware-scheduling" so its opening brace and contents align with
adjacent array items, using 8 spaces for the opening brace.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tutorials/labs/topology-aware-scheduling.md`:
- Around line 452-454: Update both Pod YAML instructions in
tutorials/labs/topology-aware-scheduling.md: at lines 452-454, tell users to
create or save the snippet as multi-gpu-pod.yaml before applying it; at lines
501-503, tell users to create or save it as single-gpu-pod.yaml before applying
it.
- Line 615: Add a single trailing newline to the end of
tutorials/labs/topology-aware-scheduling.md, preserving all existing content,
and run the relevant markdownlint or repository health checks to confirm MD047
passes.

---

Nitpick comments:
In `@sidebars-tutorials.js`:
- Around line 54-58: Adjust the indentation of the sidebar item block beginning
with the object containing id "labs/topology-aware-scheduling" so its opening
brace and contents align with adjacent array items, using 8 spaces for the
opening brace.

In `@tutorials/labs/topology-aware-scheduling.md`:
- Line 20: Split the combined imports in the MDX file into separate lines,
keeping the existing Tabs and TabItem imports unchanged and preserving their
order.
- Around line 485-486: Replace the manual pod watch with kubectl wait using a
60-second timeout: update tutorials/labs/topology-aware-scheduling.md lines
485-486 for pod/multi-gpu and lines 534-535 for pod/single-gpu, preserving the
subsequent annotation commands.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7ffda528-5b1a-4cba-8197-39305e35e66b

📥 Commits

Reviewing files that changed from the base of the PR and between 397c662 and 84699f9.

📒 Files selected for processing (2)
  • sidebars-tutorials.js
  • tutorials/labs/topology-aware-scheduling.md

Comment thread tutorials/labs/topology-aware-scheduling.md
Comment thread tutorials/labs/topology-aware-scheduling.md Outdated
@hami-robot

hami-robot Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@hami-robot hami-robot Bot removed the lgtm label Jul 15, 2026
@maishivamhoo123 maishivamhoo123 force-pushed the feat/lab3-topology-scheduling branch from 547e688 to 6e7e560 Compare July 15, 2026 06:10
Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
@maishivamhoo123 maishivamhoo123 force-pushed the feat/lab3-topology-scheduling branch from 6e7e560 to 1a8f060 Compare July 15, 2026 06:13
@rootsongjc

Copy link
Copy Markdown
Contributor

@maishivamhoo123 can you change the lab number to 10, because there was another lab 9 in progress. #608

@maishivamhoo123 maishivamhoo123 changed the title docs: add lab 9 topology-aware scheduling tutorial docs: add lab 10 topology-aware scheduling tutorial Jul 15, 2026
Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
{
"op": "add",
"path": "/spec/template/spec/containers/1/command/-",
"value": "--sgpu-topology-aware=true"

@mesutoezdil mesutoezdil Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this flag does not exist on the hami scheduler, sgpu-topology-aware is a metax device plugin flag, unrelated to nvidia topology scoring?

Apply the Pod:

```bash
kubectl apply -f multi-gpu-pod.yaml

@mesutoezdil mesutoezdil Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multi-gpu-pod.yaml is never created anywhere in this lab, so this command fails with file not found?

Apply the Pod:

```bash
kubectl apply -f single-gpu-pod.yaml

@mesutoezdil mesutoezdil Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

single-gpu-pod.yaml has the same problem, it is never created before this apply command?

git clone https://github.com/Project-HAMi/HAMi.git
cd HAMi
git log --until=2026-07-14 --oneline -1 # should output 5dca58e
git checkout 5dca58e # or the commit hash you just got

@mesutoezdil mesutoezdil Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this commit is from 2026-06-11, but git log --until=2026-07-14 on the real hami repo returns a different, newer commit from 2026-07-13?

@maishivamhoo123 maishivamhoo123 Jul 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2026-07-15 155511

and if this is different also then there is no problem we can use this same . because this is working on this branch only.

@mesutoezdil

mesutoezdil commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

this lab has no chinese translation, every other lab in this repo has one under i18n/zh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add New Tutorial: GPU Topology-Aware Scheduling on Fake GPUs

3 participants