Provide the V1 tokenizer inventory activation bridge

[i386]

Purpose

Provide the V1 native-serving-plugin activation contract for tokenizer-aware serving plugins.

Design

• Mesh materializes the complete native vocabulary while binding the source GGUF.
• ActivationContext carries a borrowed TokenizerInventoryView with native token IDs, byte pieces, and control-token identities.
• The host-owned view is valid for the activation call. A plugin copies or transforms the vocabulary before activate returns.
• The V1 plugin table is exported as mesh_native_serving_plugin_v1.
• Mesh retains ownership of model execution, verification, lifecycle ordering, and the absolute proposal deadline.

Use

A tokenizer-aware plugin can build an immutable native-token table during activation. Vocabulary discovery and table construction therefore complete before the proposal/decode path.

Dependency

This draft is based on #1149, which provides the native serving-plugin host and generation lifecycle.

Validation

• just with-lld cargo test -p model-artifact -p mesh-native-serving-plugin-host -p skippy-server
• just with-lld cargo clippy -p model-artifact -p mesh-native-serving-plugin-host -p skippy-server -- -D warnings

Summary by CodeRabbit

• New Features

  • Native serving plugins can access the model’s complete tokenizer inventory during activation.
  • Tokenizer inventories preserve raw byte tokens and identify control tokens.
  • GGUF model metadata can be scanned to build tokenizer inventories automatically.

• Bug Fixes

  • Added validation for unsupported, malformed, incomplete, or oversized tokenizer data.
  • Invalid tokenizer metadata now fails safely instead of producing incomplete inventories.

• Documentation

  • Documented tokenizer inventory availability and activation-time usage constraints.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 931a121b-058c-4645-ab7e-f07c61e908c5

📥 Commits

Reviewing files that changed from the base of the PR and between 5c33867 and a5a42a8.

📒 Files selected for processing (1)

• crates/skippy-server/src/tokenizer.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• crates/skippy-server/src/tokenizer.rs

---

📝 Walkthrough

Walkthrough

The change scans GGUF tokenizer data, stores the converted inventory in TokenizerCapability, and passes a borrowed ABI view to native plugins during activation.

Changes

Tokenizer inventory flow

Layer / File(s)	Summary
Tokenizer inventory API contract crates/mesh-native-serving-plugin-api/src/lib.rs, crates/mesh-native-serving-plugin-api/README.md	Defines owned inventory types, ABI-compatible entries and views, and the ActivationContext inventory pointer. Documents the call-scoped activation contract.
GGUF tokenizer inventory scanner crates/model-artifact/src/gguf.rs	Scans GGUF token arrays, preserves raw bytes, identifies control tokens, and rejects malformed or unsupported data.
Tokenizer capability inventory binding crates/skippy-server/Cargo.toml, crates/skippy-server/src/tokenizer.rs	Builds an optional inventory during stage-0 initialization, resolves GGUF paths, converts token pieces, validates IDs and conversions, and exposes an immutable accessor.
Activation inventory view wiring crates/mesh-native-serving-plugin-host/src/lib.rs, crates/mesh-native-serving-plugin-host/README.md	Converts the capability inventory into activation-owned ABI storage and passes a borrowed view to native plugins.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TokenizerCapability
  participant ModelServingHooksFactory
  participant ActivationInventory
  participant NativePlugin
  TokenizerCapability->>ModelServingHooksFactory: provide optional tokenizer inventory
  ModelServingHooksFactory->>ActivationInventory: convert inventory to ABI entries
  ActivationInventory->>NativePlugin: pass borrowed TokenizerInventoryView

Loading

Possibly related PRs

• Mesh-LLM/mesh-llm#1147: Extends TokenizerCapability with related tokenizer serving behavior.

Suggested labels: experimental

Suggested reviewers: ndizazzo, michaelneale

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the V1 tokenizer inventory activation bridge added by the pull request.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/rosetta-tokenizer-inventory-upstream

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh-native-serving-plugin-api/src/lib.rs`:
- Line 193: Preserve the existing V1 ActivationContext field order by removing
tokenizer_inventory from its current position before the legacy fields and
appending it after monotonic_now_ns. Keep all existing V1 members at their
original offsets; only add the new field as a trailing extension, unless this
change is explicitly converted to a separate V2 table.

In `@crates/skippy-server/src/tokenizer.rs`:
- Around line 189-196: Update source_gguf_path to validate each candidate path
independently and return the first one that is an existing file, allowing
model_path to be selected when source_model_path is unavailable. Add a
regression test covering an invalid source_model_path alongside a valid
model_path and verify the valid fallback is returned.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 099a90fd-c8b4-4632-a08b-443df9dd1f55

📥 Commits

Reviewing files that changed from the base of the PR and between 4bd1453 and fac223c.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (7)

• crates/mesh-native-serving-plugin-api/README.md
• crates/mesh-native-serving-plugin-api/src/lib.rs
• crates/mesh-native-serving-plugin-host/README.md
• crates/mesh-native-serving-plugin-host/src/lib.rs
• crates/model-artifact/src/gguf.rs
• crates/skippy-server/Cargo.toml
• crates/skippy-server/src/tokenizer.rs

[ndizazzo]

@coderabbitai what does exposing the tokenizer allow plugins to potentially do / add as features? Can you provide some examples along with your summary?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@ndizazzo The tokenizer inventory lets a serving plugin reason about the model’s actual native token space.

A plugin receives:

• Native token IDs.
• The exact raw bytes for normal token pieces.
• Explicit identities for control tokens.
• The complete vocabulary for the GGUF-bound model.

This prevents a plugin from assuming that a token ID, token boundary, or special token has the same meaning across models.

Potential plugin features include:

1. Tokenizer-correct constrained proposals
   A plugin can propose only tokens that keep output valid for a required format, such as JSON, a tool-call envelope, a prefix, or a domain-specific grammar. It can evaluate the exact bytes that each candidate token adds.

2. Reliable stop-sequence handling
   A plugin can detect whether a proposed token completes a byte-level stop sequence, including cases where the sequence spans multiple tokens. This is more reliable than matching decoded text after the fact.

3. Vocabulary-aware speculative decoding
   A plugin can build proposal logic around token IDs that are valid for the bound model. It can avoid proposals that depend on an incompatible tokenizer or an assumed vocabulary layout.

4. Token-level prefix and suffix matching
   A plugin can identify token sequences for fixed prompts, response prefixes, delimiters, or cached content. This can support reuse and proposal strategies that depend on exact token boundaries.

5. Control-token-safe behavior
   A plugin can recognize control tokens separately from ordinary byte pieces. It can avoid treating special tokens as user-visible text or accidentally emitting them in a proposal path.

6. Tokenizer-specific optimizations
   A plugin can precompute data structures during activation, such as byte-prefix indexes, delimiter-token sets, or token classes. The plugin can then use those structures during proposal without retaining host-owned activation memory.

PR summary

PR #1168 adds the V1 tokenizer activation bridge for native serving plugins.

During activation, Mesh scans the GGUF tokenizer vocabulary and creates an immutable inventory. Mesh then passes a borrowed TokenizerInventoryView through ActivationContext. The view contains token IDs, raw token bytes, and control-token information.

Plugins must copy or transform the inventory before activate returns. Mesh keeps ownership of model execution, proposal verification, lifecycle order, and the proposal deadline.

_{You are interacting with an AI system.}

[ndizazzo]

I am detecting hints of 💵 〰️ features

[i386]

I am detecting hints of 💵 〰️ features

👀