Nysa 3D occupancy - #17
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad041befea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| h, w = img.shape[:2] | ||
| p = cv2.copyMakeBorder(img, 0, max_h - h, 0, max_w - w, cv2.BORDER_REPLICATE) | ||
| padded.append(p) | ||
| inputs = depth_processor(images=padded, return_tensors="pt").to(device) |
There was a problem hiding this comment.
load_depth_model converts the Depth-Anything model to half() on CUDA, but run_depth_batch forwards depth_processor(...).to(device) without converting pixel_values to fp16. On GPU runs (including the provided Slurm config with --device cuda), this can trigger dtype mismatch errors at inference time (Float inputs vs Half weights) and stop occupancy-label generation.
Useful? React with 👍 / 👎.
| from point_cloud_gpu import ( | ||
| BEV_RANGE, | ||
| backproject_depth_to_points, | ||
| points_to_vehicle_frame, | ||
| undistort_image, |
There was a problem hiding this comment.
This script imports backproject_depth_to_points, points_to_vehicle_frame, and undistort_image from point_cloud_gpu, but that module is not present in this repository tree, so a fresh checkout cannot execute occupancy generation and will fail at import time before any work starts.
Useful? React with 👍 / 👎.
| train_dataset = WaymoE2E( | ||
| indexFile="index_train.pkl", | ||
| data_dir=args.data_dir, | ||
| n_items=250_000, |
There was a problem hiding this comment.
train_occ.py defaults to loading 250k/20k Waymo samples, but the provided OCC-generation Slurm scripts only schedule 50k train and 10k val indices; with loader_occ raising on missing occ_*.npy, the default training workflow will crash once it reaches unlabeled indices. Either generate full coverage or lower n_items defaults to the generated range.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR adds an occupancy pseudo-labeling pipeline (SegFormer + Depth-Anything backprojection + ray-cast free-space + voxelization), along with training entrypoints for baseline/OG/occupancy variants and several visualization + SLURM utilities to sanity-check outputs.
Changes:
- Add offline occupancy pseudo-label generation and 2D/3D visualization scripts.
- Add new training scripts/SLURM launchers for baseline, OG, and occupancy training.
- Introduce occupancy-aware model/head + Lightning training logic, plus loaders and sanity-check scripts.
Reviewed changes
Copilot reviewed 23 out of 26 changed files in this pull request and generated 25 comments.
Show a summary per file
| File | Description |
|---|---|
| src/camera-based-e2e/viz_seg.py | Runs SegFormer on extracted frames and writes side-by-side segmentation outputs. |
| src/camera-based-e2e/viz_occ.py | Matplotlib-only 2D occupancy grid sanity visualization + class counts. |
| src/camera-based-e2e/viz_occ_3d.py | Plotly 3D scatter visualization of occupancy voxels. |
| src/camera-based-e2e/train_og.slurm | SLURM launcher for OG training. |
| src/camera-based-e2e/train_og.py | PyTorch Lightning training entrypoint for OG model setup/training. |
| src/camera-based-e2e/train_occ.slurm | SLURM launcher for occupancy training. |
| src/camera-based-e2e/train_occ.py | Lightning training entrypoint for occupancy (loads OCC labels). |
| src/camera-based-e2e/train_baseline.slurm | SLURM launcher for baseline training. |
| src/camera-based-e2e/train_baseline.py | Lightning training entrypoint for baseline variant. |
| src/camera-based-e2e/Point_cloud/viz_seg.py | Duplicate SegFormer visualization script under Point_cloud. |
| src/camera-based-e2e/Point_cloud/vim_occ.py | Duplicate occupancy visualizer under Point_cloud (currently misnamed). |
| src/camera-based-e2e/Point_cloud/vim_occ_3d.py | Duplicate 3D occupancy visualizer under Point_cloud (currently misnamed). |
| src/camera-based-e2e/Point_cloud/create_occ_val.slurm | SLURM array job for generating val occupancy labels. |
| src/camera-based-e2e/Point_cloud/create_occ_train.slurm | SLURM array job for generating train occupancy labels. |
| src/camera-based-e2e/Point_cloud/create_occ_labels.py | Main offline pseudo-label generator (seg+depth→points→raycast→voxels). |
| src/camera-based-e2e/models/monocular_occ.py | Model variant that adds an occupancy head/logits output. |
| src/camera-based-e2e/models/monocular_baseline.py | Baseline model variant updates (depth supervision + proposal scoring). |
| src/camera-based-e2e/models/base_model_occ.py | LightningModule logic for ADE/scoring/depth + occupancy CE loss. |
| src/camera-based-e2e/models/base_model_baseline.py | LightningModule logic for ADE/scoring/depth losses (baseline). |
| src/camera-based-e2e/loader_occ.py | Dataset loader that optionally loads occupancy grids from disk. |
| src/camera-based-e2e/loader_baseline.py | Dataset loader for baseline training. |
| src/camera-based-e2e/check_past_stats.py | Utility script to inspect past-state stats for normalization debugging. |
| src/camera-based-e2e/check_occ.py | Utility script to sanity-check OCC grid existence/shape/dtype/values. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| from protos import e2e_pb2 | ||
| from point_cloud_gpu import ( | ||
| BEV_RANGE, | ||
| backproject_depth_to_points, | ||
| points_to_vehicle_frame, | ||
| undistort_image, | ||
| ) |
There was a problem hiding this comment.
create_occ_labels.py imports point_cloud_gpu (and symbols like undistort_image), but there is no point_cloud_gpu.py (or package) under src/camera-based-e2e / src/camera-based-e2e/Point_cloud in this PR, so the script will fail immediately with ModuleNotFoundError. Add the missing module to the repo (or change the import to the correct existing module path) and ensure the SLURM PYTHONPATH setup matches that location.
| unique_vox = np.unique(valid_flat) | ||
| view_counts[unique_vox] += 1 | ||
|
|
||
| # Fflter each camera's points — dynamic objects bypass the view count check |
There was a problem hiding this comment.
Typo in comment: "Fflter" should be "Filter".
| fused_pts = torch.cat(all_pts, dim=0) | ||
| fused_labels = torch.cat(all_labels, dim=0) | ||
|
|
||
| # ray cast on blank gris -> free space |
There was a problem hiding this comment.
Typo in comment: "gris" should be "grid".
| train_loader = torch.utils.data.DataLoader( | ||
| train_dataset, | ||
| batch_size=args.batch_size, | ||
| num_workers=4, | ||
| collate_fn=collate_with_images, |
There was a problem hiding this comment.
nw is computed above based on dataset choice, but the DataLoader here hard-codes num_workers=4, making nw unused and preventing the intended per-dataset worker setting from taking effect. Use num_workers=nw (or remove nw entirely if the constant value is intentional).
| train_loader = torch.utils.data.DataLoader( | ||
| train_dataset, | ||
| batch_size=args.batch_size, | ||
| num_workers=4, | ||
| collate_fn=collate_with_images, |
There was a problem hiding this comment.
nw is set based on dataset choice but the DataLoader hard-codes num_workers=4, which makes nw unused and can unintentionally change performance/behavior for the Waymo vs NuScenes configs. Either wire num_workers=nw or remove the unused nw variable if a constant is desired.
| class HomogeneousConcatBatchSampler(BatchSampler): | ||
| """Emit batches from one ConcatDataset source at a time.""" | ||
|
|
||
| def __init__( | ||
| self, | ||
| dataset_lengths: tuple[int, int], | ||
| batch_size: int, | ||
| rank: int | None = None, | ||
| world_size: int | None = None, | ||
| drop_last: bool = False, | ||
| shuffle: bool = True, | ||
| seed: int = 42, | ||
| source_ratio: tuple[int, int] = (1, 1), | ||
| **kwargs, |
There was a problem hiding this comment.
HomogeneousConcatBatchSampler is duplicated here (and again in train_baseline.py / train_occ.py), but an equivalent implementation already exists in src/camera-based-e2e/train.py (around lines 27+). Consider importing it from a shared module (e.g., utils/samplers.py) to avoid future drift/bugfixes needing to be applied in multiple places.
| PYTHON=/scratch/gilbreth/kumar753/conda_envs/robo_env_310_new/bin/python | ||
| SCRIPT_DIR=/scratch/gilbreth/kumar753/robotvision/robotvision/src/camera-based-e2e | ||
| DATA_DIR=/scratch/gilbreth/kumar753/robotvision/waymo_end_to_end_camera_v1_0_0/waymo_open_dataset_end_to_end_camera_v_1_0_0 | ||
| OCC_ROOT=/scratch/gilbreth/kumar753/waymo_occ_new |
There was a problem hiding this comment.
This SLURM script hard-codes absolute paths for the Python environment, code checkout, dataset, and OCC root. Consider parameterizing via env vars/args (or documenting required variables) so others can run it without editing a tracked file.
| @@ -0,0 +1,36 @@ | |||
| #!/bin/bash | |||
| #SBATCH --job-name=waymo_occ_test | |||
There was a problem hiding this comment.
The SLURM job name is waymo_occ_test, but this script is for generating train occupancy labels (--split train). Consider renaming the job (--job-name) to match the actual workload for easier queue monitoring/log triage.
| from typing import Optional | ||
| import random | ||
|
|
||
| devices = ['cuda:0', 'cuda:1'] |
There was a problem hiding this comment.
devices = ['cuda:0', 'cuda:1'] is defined but never used. Remove it (or use it) to avoid dead code and confusion about multi-GPU behavior in the dataset.
| from typing import Optional | ||
| import random | ||
|
|
||
| devices = ['cuda:0', 'cuda:1'] |
There was a problem hiding this comment.
devices = ['cuda:0', 'cuda:1'] is unused. Consider removing it to avoid misleading readers into thinking the loader does device placement / multi-GPU selection.
No description provided.