Transformer specification and auto-generation method for the existing models

[aftersnow]

Feature request:

Transformer is the dominant architecture for modern LLMs, and its design has largely converged. For example, most state-of-the-art open-source models adopt GQA/MLA for the Attention layer and MoE for the MLP layer. As a result, a Transformer can be viewed as a composition of standardized building blocks. This enables further abstraction of a unified architectural specification across different open-source models, which can serve as the Transformer specification in ModelPack. Based on this specification, many valuable capabilities become possible.

Expected Outcome:

• Jointly complete a unified Transformer specification (an in-progress PR already exists)
• Using vLLM and SGLang, conduct POCs on three or more mainstream open-source Transformer models based on this specification
• Design a workflow or Claude skills that can automatically generate Transformer specification definitions from models in the Hugging Face transformers repository

Use case:

For inference engines, it enables automatic support for multiple Transformer models, so newly trained Transformer models can be supported without per-model adaptation.

[obsidiannnn]

Hi @aftersnow , thanks for opening this — this is a really interesting direction.

I’ve read through the issue and I’m currently reviewing the in-progress PR for the unified Transformer specification.

I’d like to contribute by focusing on auto-generating Transformer specification definitions from Hugging Face transformers models, starting with a few mainstream architectures (e.g., LLaMA/Mistral-style GQA, MoE-based MLPs).

Concretely, my plan would be:

Study how HF model configs map to the proposed Transformer spec

Design a small workflow/tool that extracts these components automatically

Validate the generated specs on at least one inference engine (vLLM or SGLang)

Please let me know if this scope sounds reasonable or if you’d prefer me to start with a narrower POC.

[yashasp322-ship-it]

Hi @aftersnow, thanks for opening this issue.

I’ve been experimenting with automatically generating a Transformer specification from Hugging Face models.

As a small prototype, I created a script that:

• loads a Hugging Face model configuration
• extracts key architecture parameters (layers, hidden size, attention heads, etc.)
• detects attention type (e.g., MHA vs GQA)
• identifies MoE-related settings if present
• outputs a structured Transformer specification in JSON format

I tested this on a few models such as:

• meta-llama/Llama-2-7b-hf
• mistralai/Mistral-7B-v0.1

Before opening a PR, I wanted to ask whether it would make sense to include a prototype tool like this in the repository (for example under a tools/ directory), or if there is a preferred approach for generating these specifications.

Happy to adjust the approach based on feedback.

[yashasp322-ship-it]

Update:

I implemented a small prototype for automatically generating
Transformer specifications from HuggingFace model configs.

Repository:
https://github.com/yashasp322-ship-it/hf-transformer-spec-generator

Currently tested with:

• mistralai/Mistral-7B-v0.1
• tiiuae/falcon-7b
• EleutherAI/gpt-neox-20b

Happy to adapt this to ModelPack’s transformer specification format
or extend it to support additional architectures.

[shekhar871]

Hi @aftersnow — I'm Shekhar, a first-year AI/ML student planning to apply for GSoC 2026. I've been reviewing the existing PRs on the spec and the HF config extractor work.
I'd like to focus my proposal on the parts less covered so far — specifically the end-to-end POC using vLLM and SGLang across 3+ mainstream models, and the auto-generation workflow/Claude skills component. Would this scope make sense as a GSoC project? Happy to discuss further before I submit.