utils.dist.all_gather_object()

[shahelaojieraozhi]

Hi! Dear Edward Fish:
I got a bug : module 'deepspeed.comm' has no attribute 'all_gather_object'.
If I'm not mistaken, after correctly installing DeepSpeed, I need to call DeepSpeed.comm. However, it appears that it does not have an all_gather_object method.

[ed-fish]

Hi I'm not able to reproduce this error. Can you check your dependencies against the following:

torch version .................... 2.6.0+cu124
deepspeed info ................... 0.16.3, unknown, unknown
torch cuda version ............... 12.4
torch hip version ................ None
nvcc version ..................... 12.4
deepspeed wheel compiled w. ...... torch 2.6, cuda 12.4

You can check easily using the command ds_report.

Thanks

[shahelaojieraozhi]

Hi! Dear Edward Fish:
Thanks for your kind response!
I created the environment using the command conda env create -f env.yaml
However, when I run ds_report, the logs show:

DeepSpeed general environment info:
torch install path ............... ['/home/xxxx/anaconda3/envs/Geo-Sign/lib/python3.9/site-packages/torch']
torch version .................... 2.6.0+cu124
deepspeed install path ........... ['/home/xxx/anaconda3/envs/Geo-Sign/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.16.3, unknown, unknown
torch cuda version ............... 12.4
torch hip version ................ None
nvcc version ..................... 12.4
deepspeed wheel compiled w. ...... torch 0.0, cuda 0.0

deepspeed wheel compiled w. ...... torch 0.0, cuda 0.0
Could this be the cause of the issue? Do you have any suggested solutions?

Thanks

[ed-fish]

Yes I think so. It seems your install did not compile the C++/CUDA extensions.

It could be that the current dependencies had deepspeed installed before PyTorch. I've changed the order in the dependencies now and perhaps that will work.

Otherwise:

pip uninstall deepspeed
sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install python3.9-dev 
pip install deepspeed --verbose --no-cache-dir

Then check for errors in the deepspeed install. I noticed there are a ton of dependencies in the requirements file that may not be needed - apologies for the bloat. I will deal with that when I get a moment.

[shahelaojieraozhi]

Dear Edward Fish,

Thank you again for your prompt reply.

After compiling the C++/CUDA extensions following your instructions, it was successfully installed. as shown below:

torch install path ............... ['/home/xxx/anaconda3/envs/Geo-Sign/lib/python3.9/site-packages/torch']
torch version .................... 2.6.0+cu124
deepspeed install path ........... ['/home/xxx/anaconda3/envs/Geo-Sign/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.17.5, unknown, unknown
torch cuda version ............... 12.4
torch hip version ................ None
nvcc version ..................... 12.4
deepspeed wheel compiled w. ...... torch 2.6, cuda 12.4
shared memory (/dev/shm) size .... 251.84 GB

However, a new issue has arisen: the GPU utilization remains at 100%, but the training process has stalled.

I believe this may be related to GPU communication during the DeepSpeed training process. Do you have any suggested solutions?

Thanks

[ed-fish]

Glad to hear the initial error is fixed. Can you provide some additional details, for example the type/number of GPU's, the output log, and batch size.

Does it run correctly on just one gpu with a small batch-size of maybe 2 or 4?

[shahelaojieraozhi]

GPUs: RTX A5000 24GB × 6.
When I use a single GPU, everything runs normally. However, when I configure the training to use more than one GPU, the process
stalls.
Here, I did a quick test for debugging, as follows:
`

    print("0")
    utils.dist.all_gather_object(gathered_preds_obj, tgt_pres_text_local)
    utils.dist.all_gather_object(gathered_refs_obj, tgt_refs_text_local)
    print("1")

`
After printing 0, the training stops and never proceeds to print 1.

Do you have any suggestions on how I can resolve this issue?

Thanks

[ed-fish]

Can you try with NCCL debugging active?

NCCL_DEBUG=INFO deepspeed --num_gpus=6 fine_tuning.py 

[shahelaojieraozhi]

When I set NCCL_DEBUG=INFO, I get the following logs :

WandB initialized for run: CSL_Daily_SLT-hyp_True-20250916-1956 (ID: kmfyfd7e)
Full args: Namespace(batch_size=8, epochs=40, output_dir='out/stage3_train_debug', seed=42, num_workers=8, eval=False, quick_break=-1, hidden_dim=768, gcn_out_dim=256, finetune='checkpoints/pretraining.pth', opt='AdamW', opt_eps=1e-08, opt_betas=[0.9, 0.999], weight_decay=0.01, momentum=0.9, sched='cosine', lr=0.0001, warmup_epochs=5, min_lr=1e-06, max_length=256, dataset='CSL_Daily', task='SLT', label_smoothing=0.2, pin_mem=True, max_eval_samples=1000, save_batch_name='testing', world_size=2, dist_url='env://', local_rank=0, deepspeed=False, save_one_batch=False, deepspeed_config=None, offload=False, dtype='bf16', zero_stage=0, gradient_accumulation_steps=8, gradient_clipping=1.0, use_hyperbolic=True, hyp_dim=256, init_c=1.0, hyp_lr=0.001, hyp_stabilize=100, alpha_reg=0.01, alpha=0.7, label_smoothing_hyp=0.2, hyp_text_emb_src='decoder', hyp_text_cmp='token', manual_grad_clip=False, clip_grad_norm_euclid=1.0, clip_grad_norm_hyp=0.1, load_checkpoint_dir='', num_beams=4, max_tgt_len=100, wandb=True, rgb_support=False, wandb_project='Uni-Sign-Hyperbolic-v4', wandb_run_name=None, rank=0, gpu=0, distributed=True, dist_backend='nccl', wandb_run=<wandb.sdk.wandb_run.Run object at 0x7948fa40a700>)
CLI init_c = 1.0
Creating datasets...
Train dataset size: 64
Dev dataset size: 64
Test dataset size: 64
Dataloaders created.
Gradient accumulation steps: 8
Steps per epoch: 1
Total training steps calculated: 40
Creating Uni-Sign model...
***********************************
Loading Model Checkpoint for Finetuning: checkpoints/pretraining.pth
***********************************
  » Skipping manifold curvature parameter during finetune load: manifold.isp_c
  » Skipping manifold curvature parameter during finetune load: hyp_proj_body.manifold.isp_c
  » Skipping manifold curvature parameter during finetune load: hyp_proj_left.manifold.isp_c
  » Skipping manifold curvature parameter during finetune load: hyp_proj_right.manifold.isp_c
  » Skipping manifold curvature parameter during finetune load: hyp_proj_face.manifold.isp_c
  » Skipping manifold curvature parameter during finetune load: hyp_proj_text.manifold.isp_c
  » Skipping manifold curvature parameter during finetune load: geom_loss.manifold.isp_c
Missing keys during finetune load:
 hyp_attn_W
hyp_attn_b
global_step
manifold.isp_c
hyp_proj_body.manifold.isp_c
hyp_proj_right.manifold.isp_c
hyp_proj_left.manifold.isp_c
hyp_proj_face.manifold.isp_c
hyp_proj_text.manifold.isp_c
geom_loss.margin_base
geom_loss.manifold.isp_c
Unexpected keys during finetune load:
 geom_loss.margin_scale
Model created/loaded. Number of parameters: 588.27M
Hyperbolic branch is active. Setting up Riemannian and Euclidean optimizers...
Found 2 hyperbolic and 460 Euclidean parameters.
Euclidean Optimizer: AdamW (
Parameter Group 0
    amsgrad: False
    betas: [0.9, 0.999]
    capturable: False
    differentiable: False
    eps: 1e-08
    foreach: None
    fused: None
    lr: 0.0001
    maximize: False
    weight_decay: 0.01
)
Hyperbolic Optimizer: RiemannianAdam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    capturable: False
    differentiable: False
    eps: 1e-08
    foreach: None
    fused: None
    lr: 0.001
    maximize: False
    stabilize: 100
    weight_decay: 0.0
)
LR Scheduler (cosine) configured for main optimizer.
Initializing DeepSpeed...
--- Generated DeepSpeed Config ---
{
  "train_batch_size": 128,
  "train_micro_batch_size_per_gpu": 8,
  "steps_per_print": 10,
  "gradient_accumulation_steps": 8,
  "zero_optimization": {
    "stage": 0,
    "offload_param": {
      "device": "none",
      "pin_memory": true
    },
    "offload_optimizer": {
      "device": "none",
      "pin_memory": true
    },
    "stage3_param_persistence_threshold": 10000.0,
    "stage3_max_live_parameters": 30000000.0,
    "stage3_prefetch_bucket_size": 30000000.0,
    "memory_efficient_linear": false,
    "contiguous_gradients": true,
    "overlap_comm": true
  },
  "gradient_clipping": 1.0,
  "prescale_gradients": false,
  "wall_clock_breakdown": false,
  "hybrid_engine": {
    "enabled": false,
    "max_out_tokens": 512,
    "inference_tp_size": 1,
    "release_inference_cache": false,
    "pin_parameters": true,
    "tp_gather_partition_size": 8
  },
  "tensorboard": {
    "enabled": false,
    "output_path": "/ds_tensorboard_logs/",
    "job_name": "_tensorboard"
  },
  "bfloat16": {
    "enabled": true
  }
}
---------------------------------
[Rank 0] Initializing DeepSpeed engine...
[2025-09-16 19:57:07,627] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed info: version=0.17.5, git-hash=unknown, git-branch=unknown
[2025-09-16 19:57:07,628] [INFO] [config.py:684:__init__] Config mesh_device None world_size = 2
[2025-09-16 19:57:07,654] [INFO] [engine.py:1356:_configure_distributed_model] ********** distributed groups summary **********
	 self.dp_world_size=2
	 self.mp_world_size=1
	 self.seq_dp_world_size=2
	 self.sequence_parallel_size=1
***********************************************
[2025-09-16 19:57:07,778] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2025-09-16 19:57:07,779] [INFO] [logging.py:107:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2025-09-16 19:57:07,779] [INFO] [logging.py:107:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-09-16 19:57:07,807] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW
[2025-09-16 19:57:07,807] [INFO] [logging.py:107:log_dist] [Rank 0] Creating BF16 optimizer
[2025-09-16 19:57:08,053] [INFO] [utils.py:781:see_memory_usage] begin bf16_optimizer
[2025-09-16 19:57:08,054] [INFO] [utils.py:782:see_memory_usage] MA 1.13 GB         Max_MA 2.54 GB         CA 2.65 GB         Max_CA 3 GB
[2025-09-16 19:57:08,054] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 37.81 GB, percent = 7.5%
[2025-09-16 19:57:08,216] [INFO] [utils.py:781:see_memory_usage] before initializing group 0
[2025-09-16 19:57:08,217] [INFO] [utils.py:782:see_memory_usage] MA 1.13 GB         Max_MA 1.13 GB         CA 2.65 GB         Max_CA 3 GB
[2025-09-16 19:57:08,218] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 37.83 GB, percent = 7.5%
[2025-09-16 19:57:08,389] [INFO] [utils.py:781:see_memory_usage] after initializing group 0
[2025-09-16 19:57:08,390] [INFO] [utils.py:782:see_memory_usage] MA 4.38 GB         Max_MA 4.38 GB         CA 7.03 GB         Max_CA 7 GB
[2025-09-16 19:57:08,390] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 37.88 GB, percent = 7.5%
[2025-09-16 19:57:08,557] [INFO] [utils.py:781:see_memory_usage] end bf16_ optimizer
[2025-09-16 19:57:08,559] [INFO] [utils.py:782:see_memory_usage] MA 4.38 GB         Max_MA 4.38 GB         CA 7.03 GB         Max_CA 7 GB
[2025-09-16 19:57:08,559] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 38.15 GB, percent = 7.6%
[2025-09-16 19:57:08,559] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Final Optimizer = BF16_Optimizer
[2025-09-16 19:57:08,559] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2025-09-16 19:57:08,559] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7948f9d69e50>
[2025-09-16 19:57:08,560] [INFO] [logging.py:107:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]]
[2025-09-16 19:57:08,561] [INFO] [logging.py:107:log_dist] [Rank 0] [TorchCheckpointEngine] Initialized with serialization = True
[2025-09-16 19:57:08,562] [INFO] [config.py:954:print] DeepSpeedEngine configuration:
[2025-09-16 19:57:08,562] [INFO] [config.py:958:print]   activation_checkpointing_config  {
    "partition_activations": false,
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "synchronize_checkpoint_boundary": false,
    "profile": false
}
[2025-09-16 19:57:08,562] [INFO] [config.py:958:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'intra_op_parallelism': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-09-16 19:57:08,562] [INFO] [config.py:958:print]   amp_enabled .................. False
[2025-09-16 19:57:08,562] [INFO] [config.py:958:print]   amp_params ................... False
[2025-09-16 19:57:08,562] [INFO] [config.py:958:print]   autotuning_config ............ {
    "enabled": false,
    "start_step": null,
    "end_step": null,
    "metric_path": null,
    "arg_mappings": null,
    "metric": "throughput",
    "model_info": null,
    "results_dir": "autotuning_results",
    "exps_dir": "autotuning_exps",
    "overwrite": true,
    "fast": true,
    "start_profile_step": 3,
    "end_profile_step": 5,
    "tuner_type": "gridsearch",
    "tuner_early_stopping": 5,
    "tuner_num_trials": 50,
    "model_info_path": null,
    "mp_size": 1,
    "max_train_batch_size": null,
    "min_train_batch_size": 1,
    "max_train_micro_batch_size_per_gpu": 1.024000e+03,
    "min_train_micro_batch_size_per_gpu": 1,
    "num_tuning_micro_batch_sizes": 3
}
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   bfloat16_config .............. enabled=True immediate_grad_update=False check_grad_overflow=False
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   checkpoint_config ............ {'tag_validation': 'WARN', 'checkpoint_serialization': True, 'writer': None}
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   checkpoint_parallel_write_pipeline  False
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   checkpoint_tag_validation_enabled  True
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   checkpoint_tag_validation_fail  False
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7948f9d984c0>
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   communication_data_type ...... None
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   compile_config ............... deepcompile=False free_activation=False offload_activation=False offload_opt_states=False double_buffer=True symmetric_memory=False debug_log=False offload_parameters=False sync_before_reduce=False sync_after_reduce=False sync_before_allgather=False sync_after_allgather=False keep_int_input_tensors=True keep_all_input_tensors=False
[2025-09-16 19:57:08,563] [INFO] [config.py:958:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   curriculum_enabled_legacy .... False
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   curriculum_params_legacy ..... False
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'pin_memory': False, 'curriculum_learning': {'enabled': False}, 'dynamic_batching': {'enabled': False, 'lr_scaling_method': 'linear', 'min_batch_size': 1, 'max_batch_size': None, 'sequence_picking_order': 'dataloader', 'verbose': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   data_efficiency_enabled ...... False
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   dataloader_drop_last ......... False
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   disable_allgather ............ False
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   dump_state ................... False
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   eigenvalue_enabled ........... False
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   eigenvalue_gas_boundary_resolution  1
[2025-09-16 19:57:08,564] [INFO] [config.py:958:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   eigenvalue_layer_num ......... 0
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   eigenvalue_max_iter .......... 100
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   eigenvalue_stability ......... 1e-06
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   eigenvalue_tol ............... 0.01
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   eigenvalue_verbose ........... False
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   elasticity_enabled ........... False
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   float16_config ............... enabled=False auto_cast=False loss_scale=0.0 initial_scale_power=16 loss_scale_window=1000 hysteresis=2 consecutive_hysteresis=False min_loss_scale=1 fp16_master_weights_and_grads=False
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   flops_profiler_config ........ {
    "enabled": false,
    "recompute_fwd_factor": 0.0,
    "profile_step": 1,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": null
}
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   global_rank .................. 0
[2025-09-16 19:57:08,565] [INFO] [config.py:958:print]   grad_accum_dtype ............. None
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   gradient_accumulation_steps .. 8
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   gradient_clipping ............ 1.0
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   gradient_predivide_factor .... 1.0
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   graph_harvesting ............. False
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   load_universal_checkpoint .... False
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   memory_breakdown ............. False
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   mics_hierarchial_params_gather  False
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   mics_shard_size .............. -1
[2025-09-16 19:57:08,566] [INFO] [config.py:958:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='/ds_tensorboard_logs/', job_name='_tensorboard') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   nebula_config ................ {
    "enabled": false,
    "persistent_storage_path": null,
    "persistent_time_interval": 100,
    "num_of_version_in_retention": 2,
    "enable_nebula_load": true,
    "load_path": null
}
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   optimizer_legacy_fusion ...... False
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   optimizer_name ............... None
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   optimizer_params ............. None
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   pld_enabled .................. False
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   pld_params ................... False
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   prescale_gradients ........... False
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   scheduler_name ............... None
[2025-09-16 19:57:08,567] [INFO] [config.py:958:print]   scheduler_params ............. None
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   seq_parallel_communication_data_type  torch.float32
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   sparse_attention ............. None
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   sparse_gradients_enabled ..... False
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   steps_per_print .............. 10
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   tensor_parallel_config ....... dtype=torch.float16 autotp_size=0 tp_overlap_comm=False tensor_parallel=TPConfig(tp_size=1, tp_grain_size=1, mpu=None, tp_group=None) injection_policy_tuple=None keep_module_on_host=False replace_with_kernel_inject=False
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   timers_config ................ enabled=True synchronized=True
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   torch_autocast_dtype ......... None
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   torch_autocast_enabled ....... False
[2025-09-16 19:57:08,568] [INFO] [config.py:958:print]   torch_autocast_lower_precision_safe_modules  None
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   train_batch_size ............. 128
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   train_micro_batch_size_per_gpu  8
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   use_data_before_expert_parallel_  False
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   use_node_local_storage ....... False
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   wall_clock_breakdown ......... False
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   weight_quantization_config ... None
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   world_size ................... 2
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   zero_allow_untested_optimizer  False
[2025-09-16 19:57:08,569] [INFO] [config.py:958:print]   zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100000000, max_in_cpu=1000000000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=True, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) zenflow=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=30000000 param_persistence_threshold=10000 model_persistence_threshold=9223372036854775807 max_live_parameters=30000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=False pipeline_loading_checkpoint=False override_module_apply=True log_trace_cache_warnings=False
[2025-09-16 19:57:08,570] [INFO] [config.py:958:print]   zero_enabled ................. False
[2025-09-16 19:57:08,570] [INFO] [config.py:958:print]   zero_force_ds_cpu_optimizer .. True
[2025-09-16 19:57:08,570] [INFO] [config.py:958:print]   zero_optimization_stage ...... 0
[2025-09-16 19:57:08,570] [INFO] [config.py:944:print_user_config]   json = {
    "train_batch_size": 128,
    "train_micro_batch_size_per_gpu": 8,
    "steps_per_print": 10,
    "gradient_accumulation_steps": 8,
    "zero_optimization": {
        "stage": 0,
        "offload_param": {
            "device": "none",
            "pin_memory": true
        },
        "offload_optimizer": {
            "device": "none",
            "pin_memory": true
        },
        "stage3_param_persistence_threshold": 1.000000e+04,
        "stage3_max_live_parameters": 3.000000e+07,
        "stage3_prefetch_bucket_size": 3.000000e+07,
        "memory_efficient_linear": false,
        "contiguous_gradients": true,
        "overlap_comm": true
    },
    "gradient_clipping": 1.0,
    "prescale_gradients": false,
    "wall_clock_breakdown": false,
    "hybrid_engine": {
        "enabled": false,
        "max_out_tokens": 512,
        "inference_tp_size": 1,
        "release_inference_cache": false,
        "pin_parameters": true,
        "tp_gather_partition_size": 8
    },
    "tensorboard": {
        "enabled": false,
        "output_path": "/ds_tensorboard_logs/",
        "job_name": "_tensorboard"
    },
    "bfloat16": {
        "enabled": true
    }
}
[Rank 0] DeepSpeed engine initialized.
DeepSpeed initialized.
[Rank 0] Starting training from epoch 0 to 39
--- Epoch 0/39 ---
/home/xxx/Geo-Sign/models.py:261: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False): # Geoopt operations in fp32
/home/xxx/Geo-Sign/models.py:309: RuntimeWarning: Token mode contrastive loss not fully shown here, using pooled as fallback for example structure.
  warnings.warn("Token mode contrastive loss not fully shown here, using pooled as fallback for example structure.", RuntimeWarning)
Epoch: [0/39]  [0/4]  eta: 0:00:25  lr: 0.000000  hyp_lr: 0.001000  loss: 4.3107  ce_loss: 5.0842  margin_loss: 2.2953  alpha: 0.723  curvature: 1.0000  time: 6.4219  data: 2.3102  max mem: 7833MB
Epoch: [0/39]  [1/4]  eta: 0:00:11  lr: 0.000000  hyp_lr: 0.001000  loss: 4.7326  ce_loss: 5.6490  margin_loss: 2.2974  alpha: 0.727  curvature: 1.0000  time: 3.7559  data: 1.1552  max mem: 7833MB
Epoch: [0/39]  [2/4]  eta: 0:00:05  lr: 0.000000  hyp_lr: 0.001000  loss: 4.9433  ce_loss: 5.9410  margin_loss: 2.2922  alpha: 0.727  curvature: 1.0000  time: 2.6669  data: 0.7701  max mem: 7968MB
Epoch: [0/39]  [3/4]  eta: 0:00:02  lr: 0.000000  hyp_lr: 0.001000  loss: 4.8553  ce_loss: 5.8006  margin_loss: 2.2932  alpha: 0.730  curvature: 1.0000  time: 2.1237  data: 0.5776  max mem: 7968MB
Epoch: [0/39] Total time: 0:00:08 (2.1762 s / it)
Epoch 0 averaged stats: lr: 0.000000  hyp_lr: 0.001000  loss: 4.8553  ce_loss: 5.8006  margin_loss: 2.2932  alpha: 0.730  curvature: 1.0000
Epoch 0 training finished. Avg Loss: 4.7852
--- Running evaluation for Epoch 0 on Rank 0 ---
[DEBUG evaluate] Step 0, Phase dev, Global args.eval: False, batch_eval_data keys: []
Eval (dev):  [0/8]  eta: 0:00:32  loss: 4.5566  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -4.3315  curvature: 1.0000  time: 4.0804  data: 2.3340  max mem: 7968MB
[DEBUG evaluate] Step 1, Phase dev, Global args.eval: False, batch_eval_data keys: []
Eval (dev):  [1/8]  eta: 0:00:19  loss: 4.5593  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.9503  curvature: 1.0000  time: 2.7319  data: 1.1671  max mem: 7968MB
[DEBUG evaluate] Step 2, Phase dev, Global args.eval: False, batch_eval_data keys: []
Eval (dev):  [2/8]  eta: 0:00:12  loss: 4.8206  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.9759  curvature: 1.0000  time: 2.0328  data: 0.7781  max mem: 7968MB
Eval (dev):  [3/8]  eta: 0:00:08  loss: 4.5170  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -4.1351  curvature: 1.0000  time: 1.7082  data: 0.5836  max mem: 7968MB
Eval (dev):  [4/8]  eta: 0:00:06  loss: 4.6437  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -4.0130  curvature: 1.0000  time: 1.6038  data: 0.4670  max mem: 7968MB
Eval (dev):  [5/8]  eta: 0:00:04  loss: 4.7461  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.8734  curvature: 1.0000  time: 1.4918  data: 0.3892  max mem: 7968MB
Eval (dev):  [6/8]  eta: 0:00:02  loss: 4.8898  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.9158  curvature: 1.0000  time: 1.4741  data: 0.3336  max mem: 7968MB
Eval (dev):  [7/8]  eta: 0:00:01  loss: 5.0261  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.9556  curvature: 1.0000  time: 1.3869  data: 0.2919  max mem: 7968MB
Eval (dev): Total time: 0:00:11 (1.4110 s / it)
0

[ed-fish]

Ah okay - I think the issue is that I was running eval on rank 0 but it should be over all ranks. Can you try the latest push of finetuning.py.

[shahelaojieraozhi]

Dear Edward Fish,

Sorry, I was too busy yesterday to check GitHub. When I run the new finetuning.py, the training process still stops, as shown below:

Eval (dev):  [4/8]  eta: 0:00:06  loss: 4.6437  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -4.0130  curvature: 1.0000  time: 1.6476  data: 0.5044  max mem: 7968MB
Eval (dev):  [5/8]  eta: 0:00:04  loss: 4.7461  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.8734  curvature: 1.0000  time: 1.5298  data: 0.4204  max mem: 7968MB
Eval (dev):  [6/8]  eta: 0:00:03  loss: 4.8898  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.9158  curvature: 1.0000  time: 1.5096  data: 0.3604  max mem: 7968MB
Eval (dev):  [7/8]  eta: 0:00:01  loss: 5.0261  bleu1: 0.00  bleu2: 0.00  bleu3: 0.00  bleu4: 0.00  rouge: 0.0000  hyp_sim_mean: -3.9556  curvature: 1.0000  time: 1.4194  data: 0.3153  max mem: 7968MB
Eval (dev): Total time: 0:00:11 (1.4428 s / it)

Evaluation (dev) gathered 128 predictions and 128 references.
{'bleu1': 20.52349791790601, 'bleu2': 11.495820209052239, 'bleu3': 7.3030585753902635, 'bleu4': 4.603435670682174}
Rough: 22.91

* Evaluation Complete (dev - Rank 0 Results):
  loss: 4.7177
  bleu1: 20.5235
  bleu2: 11.4958
  bleu3: 7.3031
  bleu4: 4.6034
  rouge: 22.9138
  hyp_sim_mean: -4.0131
  curvature: 1.0000 (last line)

Thanks

[ed-fish]

From here it looks as though the code is working correctly. Does it not progress past the development evaluation and continue training?

[shahelaojieraozhi]

Yes, when the last line is shown, the training just stops. That's so weird.

[ed-fish]

Hi sorry for the delay - I will look into this issue this weekend and be in touch with a solution.