Hi, and thanks for releasing ImDy and the pretrained checkpoints — really nice work.
I'm trying to run the released imdy_pretrain.pt zero-shot on real AddBiomechanics subjects to predict ground reaction forces, but I can't reproduce sensible predictions. I believe it's because the AddBiomechanics → model-input preprocessing is not in the repo yet (the README notes "code release is in progress"). The model side works perfectly for me — I just can't match the exact input representation.
What already works on my side
Built the model from config/IDFD_mkr.yml (after copying PAST_KF/FUTURE_KF into MODEL) and loaded imdy_pretrain.pt → load_state_dict reports 0 missing / 0 unexpected.
Forward on {'mkr': [B, M, 6, 3], 'mvel': [B, M, 6, 3]} returns grfvec (B,2,24,3), grfnorm (B,2,24,1), torvec, tornorm, contact. I reconstruct GRF as grf = grfvec * grfnorm.
I read AddBiomechanics .b3d with nimblephysics (markerObservations, processingPasses[-1].groundContactForce, getMassKg). The measured vertical GRF averages ≈ body weight, so the ground-truth side is correct.
I estimated grf mean/std from the ImDy raw_train/grf.pkl (shape [24,3]) for de-normalization.
The problem
When I feed AddBiomechanics markers (the raw experimental markers in the .b3d, mapped by name / pelvis-centered), the predicted vertical GRF barely correlates with the measured force-plate GRF (≈ 0.0–0.5 across several walking and running subjects). Changing the marker subset, adding heading canonicalization, or rescaling velocity does not help — so I suspect I'm getting the input representation wrong somewhere.
Questions
Markers. For AddBiomechanics, do you fit SMPL to the motion and sample the smpl_opensim_markerset vertices (so the model always sees SMPL-vertex markers), or do you feed the raw experimental markers stored in the .b3d? If SMPL is fit, could you share or point to that fitting step?
Canonicalization / frame. What canonicalization is applied to the marker window before the model (up-axis convention, heading alignment, centering, per-frame vs per-window)? ImDy appears to be z-up while AddBiomechanics/OpenSim is y-up — how do you align them?
Output frame. In what frame is the predicted GRF λ expressed — world, or a root/heading-canonical frame? How should it be rotated back to world to compare with force plates?
Normalization. Is the mean_std / norm_*.pkl file used for the ImDy-pretrained model available? Is grf normalized in the raw per-joint [24,3] space, or after some transform?
Velocity. How is mvel computed exactly (estimate_linear_velocity) — is it scaled to per-second, and at what frame rate (30 FPS)?
Ground-truth extraction for adb. For the adb dataset keys ['torque','grf','mkr','weight'], which nimblephysics processing pass / fields do torque and grf come from, and in what units and frame?
Is an AddBiomechanics conversion script (the adb equivalent of convert.py) planned for release?
Even a rough description of the AddBiomechanics preprocessing — or a single example converted .pkl — would fully unblock this. Thanks a lot!
Hi, and thanks for releasing ImDy and the pretrained checkpoints — really nice work.
I'm trying to run the released imdy_pretrain.pt zero-shot on real AddBiomechanics subjects to predict ground reaction forces, but I can't reproduce sensible predictions. I believe it's because the AddBiomechanics → model-input preprocessing is not in the repo yet (the README notes "code release is in progress"). The model side works perfectly for me — I just can't match the exact input representation.
What already works on my side
Built the model from config/IDFD_mkr.yml (after copying PAST_KF/FUTURE_KF into MODEL) and loaded imdy_pretrain.pt → load_state_dict reports 0 missing / 0 unexpected.
Forward on {'mkr': [B, M, 6, 3], 'mvel': [B, M, 6, 3]} returns grfvec (B,2,24,3), grfnorm (B,2,24,1), torvec, tornorm, contact. I reconstruct GRF as grf = grfvec * grfnorm.
I read AddBiomechanics .b3d with nimblephysics (markerObservations, processingPasses[-1].groundContactForce, getMassKg). The measured vertical GRF averages ≈ body weight, so the ground-truth side is correct.
I estimated grf mean/std from the ImDy raw_train/grf.pkl (shape [24,3]) for de-normalization.
The problem
When I feed AddBiomechanics markers (the raw experimental markers in the .b3d, mapped by name / pelvis-centered), the predicted vertical GRF barely correlates with the measured force-plate GRF (≈ 0.0–0.5 across several walking and running subjects). Changing the marker subset, adding heading canonicalization, or rescaling velocity does not help — so I suspect I'm getting the input representation wrong somewhere.
Questions
Markers. For AddBiomechanics, do you fit SMPL to the motion and sample the smpl_opensim_markerset vertices (so the model always sees SMPL-vertex markers), or do you feed the raw experimental markers stored in the .b3d? If SMPL is fit, could you share or point to that fitting step?
Canonicalization / frame. What canonicalization is applied to the marker window before the model (up-axis convention, heading alignment, centering, per-frame vs per-window)? ImDy appears to be z-up while AddBiomechanics/OpenSim is y-up — how do you align them?
Output frame. In what frame is the predicted GRF λ expressed — world, or a root/heading-canonical frame? How should it be rotated back to world to compare with force plates?
Normalization. Is the mean_std / norm_*.pkl file used for the ImDy-pretrained model available? Is grf normalized in the raw per-joint [24,3] space, or after some transform?
Velocity. How is mvel computed exactly (estimate_linear_velocity) — is it scaled to per-second, and at what frame rate (30 FPS)?
Ground-truth extraction for adb. For the adb dataset keys ['torque','grf','mkr','weight'], which nimblephysics processing pass / fields do torque and grf come from, and in what units and frame?
Is an AddBiomechanics conversion script (the adb equivalent of convert.py) planned for release?
Even a rough description of the AddBiomechanics preprocessing — or a single example converted .pkl — would fully unblock this. Thanks a lot!