Add timm model support with architecture-aware activation extraction

[Copilot]

Implements timm model compatibility for TVSD benchmarking. Different model architectures (ViT, Swin, ConvNeXt) output varying activation shapes that break the existing pipeline expecting uniform (batch, features) tensors.

Changes

Core implementation:

• utils/load_model.py: Enable timm imports, add optional pretrained config parameter, make RESMAX path configurable via RESMAX_PATH env var
• utils/timm_helpers.py (new): Model family detection and architecture-specific activation processing
  • ViT: Flatten (batch, tokens, embed_dim) → (batch, tokens×embed_dim)
  • Swin: Handle hierarchical token structures
  • ConvNeXt: Process stage-based outputs
  • ResNet/default: Standard flattening
• utils/hooks.py: Integrate model family detection for automatic activation shape handling

Testing:

• tests/test_timm.py (new): 16 tests covering model loading, family detection, activation extraction, and shape validation for ResNet, ViT, ConvNeXt, Swin

Documentation:

• Example configs for all 4 families in configs/examples/
• README usage guide

Usage

# configs/examples/vit_base_patch16_224.yaml
model-name: vit_base_patch16_224
model-source: timm
pretrained: true
hook-interval: 5
transform:
  - name: Resize
    size: [224, 224]
  - name: ToTensor
  - name: Normalize
    mean: [0.485, 0.456, 0.406]
    std: [0.229, 0.224, 0.225]

sbatch scripts/generate_activations.sh configs/examples/vit_base_patch16_224.yaml

Backward compatible with existing torchvision models.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• huggingface.co
  • Triggering command: /usr/bin/python python -m pytest tests/test_timm.py -v (dns block)
  • Triggering command: `/usr/bin/python python -c
    from utils.load_model import load_model
    import torch

Test loading a timm model

print('Testing timm model loading...')
model, model_name, hook_interval = load_model('configs/examples/resnet50.yaml')
print(f'��� Loaded model: {model_name} with hook_inte` (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Timm compatibility</issue_title>
<issue_description>We need to build out full compatibility with timm models. I anticipate some activation shape issues for certain model classes.</issue_description>

Comments on the Issue (you are @copilot in this section)

@c-mulliken definition of done

utils/load_model.py supports loading timm models by name via config (e.g. timm: true + model_name: "resnet50" or similar).

activation extraction works for a small, explicit suite of timm models covering the annoying families:

resnet (easy)

vit (token/cls shapes)

convnext (stages)

swin (hierarchical tokens)

tests: add a cpu-only test that:

instantiates each model

runs a dummy batch through your activation pipeline

asserts expected shapes / no crashes

docs: update README/config examples to show timm usage.

ci passes.

guardrails

do not change benchmark metrics logic unless needed

prefer minimal invasive changes

if a model can’t be supported, document it + skip with a clear reason</comment_new>

• Fixes Timm compatibility #2

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.