Skip to content

feat: tool types + registry + ToolHandler trait (PR A)#80

Merged
maralbahari merged 2 commits into
vllm-project:mainfrom
ashwing:feat/tool-types-registry
Jul 1, 2026
Merged

feat: tool types + registry + ToolHandler trait (PR A)#80
maralbahari merged 2 commits into
vllm-project:mainfrom
ashwing:feat/tool-types-registry

Conversation

@ashwing

@ashwing ashwing commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements PR A from the tool framework design:

  • ResponsesTool — replaces pub type ResponsesTool = FunctionTool alias with a #[serde(tag = "type")] tagged enum covering all five Responses API tool types (function, mcp, web_search_preview, file_search, code_interpreter). Wire-compatible: existing {"type":"function",...} requests deserialize unchanged. Marked #[non_exhaustive] for future API additions.
  • ToolRegistry — request-scoped HashMap<name → ToolEntry> built from RequestPayload.tools + any server-discovered tools. Provides gateway_owned / client_owned split for post-inference dispatch routing (PR B).
  • ToolHandler traitvalidate / discover / normalize / execute lifecycle. Send + Sync for Arc<dyn ToolHandler>. Default discover() is a no-op; execute() must be implemented per handler type.
  • FunctionHandler — trivial ToolHandler impl for type: "function" tools. Normalizes to FunctionTool, never gateway-executes (client-owned).
  • normalize pipelineResponsesTool::to_function_tool() wired into RequestPayload::to_upstream_request(). vLLM always receives Vec<FunctionTool>, never raw ResponsesTool variants. Function tools pass through unchanged; gateway-managed types (mcp, web_search_preview, file_search, code_interpreter) are dropped with a tracing::debug! until their handlers land in PR B/C.
  • From<ToolOutput> for FunctionToolResultMessage — zero-friction conversion for PR B's dispatch loop.
  • McpToolParam includes headers: Option<HashMap<String,String>> for per-server auth forwarding.

What vLLM sees after this PR

Client sends vLLM receives
{"type":"function","name":"foo",...} {"type":"function","name":"foo",...} ✅ unchanged
{"type":"mcp","server_label":"db",...} nothing — dropped until PR C
{"type":"web_search_preview"} nothing — dropped until PR C

Gateway-type tools are parsed, validated, and stored in ResponseMetadata.effective_tools for rehydration across turns — they just don't reach vLLM until real normalize() implementations land.

No execution logic in this PR: no dispatch_tools, no LoopDecision, no changes to engine.rs / rehydrate.rs / persist.rs.

Depends on: #64 (io types refactor — merged). Unblocks: PR B (dispatch), PR C (MCP handler), CustomCodexTool (#68).

Test Plan

cargo build --workspace     # 0 errors
cargo clippy --workspace --all-targets -- -D warnings  # 0 warnings
cargo test --workspace      # 246 tests pass

Key new tests in types::tools::tests:

  • All 5 ResponsesTool variants round-trip through serde
  • ToolRegistry lookup, gateway_owned / client_owned split, allowed_tools filtering, duplicate-name warning, empty-name skip
  • FunctionHandler::validate() rejects empty names
  • to_upstream_request() with mixed tools — only function tool reaches vLLM
  • All-gateway tools → tools: null (empty array never sent to vLLM)

@ashwing ashwing marked this pull request as draft June 30, 2026 02:18
@ashwing ashwing marked this pull request as ready for review June 30, 2026 02:43

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing thank you. overall tools/ module looks well-structured. I have added some inline comments and it would be great to make sure the serde shapes and normalization logic are validated against real wire payloads before we merge.

We already have multi-turn tool call cassettes at crates/agentic-core/tests/cassettes/tool_calls/multi_turn/ responses_tool_calls_3turn.yaml, responses_tool_calls_5turn.yaml, and responses_tool_calls_parallel.yaml. Each turn's request.body.tools is the exact JSON the gateway receives, so parsing it into Vec<ResponsesTool> is the same validation as a real request without needing any running server.

The existing accumulator_cassette_test.rs already has the load_turn_cassette_from helper and Turn struct that deserializes request bodies as serde_json::Value. you can reuse that infrastructure directly in a new tool_normalization_test.rs.

Specifically the tests that cover:

serde_json::from_value::<Vec<ResponsesTool>>(request.body["tools"]) parses without error for every turn across all three cassettes
Each parsed tool normalizes correctly via to_function_tool(). The Function variants produce a FunctionTool with the right name/type, non-function variants return None
ToolRegistry::build on the cassette tools produces the expected entries and lookup by name works
Full round-trip: deserialize request.body into RequestPayload, call to_upstream_request(), assert the upstream tools field only contains FunctionTool entries and none of the gateway-only types leak through

Regarding MCP related code like DiscoveredTool, the discover() method, and the MCP branch in ToolRegistry::build, please don't add stub functions we can't test. discover() always returns Ok(vec![]) and DiscoveredTool is never populated, so this is dead code we can't validate.

since MCP is not in scope for this PR, please remove DiscoveredTool, the discover() method from ToolHandler, and the discovered parameter from normalize(). We can add MCP support once we have a way to test it.

Comment thread crates/agentic-core/src/types/tools/function.rs Outdated
Comment thread crates/agentic-core/src/types/tools/function.rs Outdated
Comment thread crates/agentic-core/src/types/tools/registry.rs Outdated
ashwing added a commit to ashwing/agentic-api that referenced this pull request Jun 30, 2026
1. Move tool behaviors to src/tool/ — types/ is for wire shapes only
   (registry, handler trait, normalization, FunctionHandler now live in
   src/tool/; types/tools/ keeps only ResponsesTool + param structs)

2. Replace function_tool_from_parts(&Value) with From<&FunctionToolParam>
   for FunctionTool — typed conversion over erased JSON (api-from-not-into)

3. Remove dead MCP stubs — DiscoveredTool, discover() on ToolHandler, and
   the discovered param from normalize() removed; untestable without a real
   MCP server, will be added when MCP handler lands in PR C

4. Strip section dividers (// ----) from all files per style feedback

5. Add cassette-based tests in tool_normalization_test.rs — validates
   Vec<ResponsesTool> parsing, to_function_tool() normalization,
   ToolRegistry::build lookup, and full RequestPayload round-trip against
   real wire payloads from the openai 3turn/5turn/parallel cassettes
   (12 new tests, all passing)

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
Rebased on main after vllm-project#79 and vllm-project#81 merged.

Adds src/tool/ — the behavioral layer that complements the wire types
already in types/tools/ (merged via PR vllm-project#79):

- tool/handler.rs  — ToolHandler trait, ToolOutput, ToolError
- tool/registry.rs — ToolType, ToolEntry, ToolRegistry::build/lookup/etc
- tool/function.rs — FunctionHandler + From<&FunctionToolParam> for FunctionTool
- tool/normalize.rs — ResponsesTool::to_function_tool(), From<ToolOutput>

Also adds types/tools/ wire types (params.rs with ResponsesTool enum,
param structs, NonEmptyToolName), EmptyToolNameError, and wires normalize
into RequestPayload::to_upstream_request() so vLLM always receives
Vec<FunctionTool>.

12 cassette-based tests in tool_normalization_test.rs validate the full
pipeline against real multi-turn tool-call cassettes.

Addresses all PR vllm-project#80 review feedback:
- types/ → wire shapes only; tool/ → behaviors
- From<&FunctionToolParam> for FunctionTool (typed conversion)
- MCP registry entries deferred to PR C (discovery not yet wired)
- EmptyToolNameError in types/ (no cross-layer import)
- ToolOutput derives Debug + Clone

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
@ashwing ashwing force-pushed the feat/tool-types-registry branch from 18338b2 to 590fede Compare June 30, 2026 08:25
@ashwing

ashwing commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased on main after #79 and #81 merged. Squashed into a single clean commit — conflicts resolved, all 264 tests pass.

@ashwing

ashwing commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author

Done across all commits on this rebase. Specifically:

1. Section dividers — stripped from all files.

2. From<&FunctionToolParam> for normalization — replaced the manual Value::get() extraction path with impl From<&FunctionToolParam> for FunctionTool. FunctionHandler::normalize now deserializes the stored &Value back into FunctionToolParam and calls FunctionTool::from(&p) — single conversion path, no divergence risk.

3. types/ vs tool/ — agreed on the distinction. Wire shapes (ResponsesTool, param structs, NonEmptyToolName) stay in types/tools/. The behavioral layer (ToolRegistry, ToolHandler, FunctionHandler, normalize pipeline) moved to src/tool/. types/tools/mod.rs doc comment now explicitly states it contains only serde shapes.

4. Cassette tests + MCP stubs removed — added tool_normalization_test.rs with 12 tests against the openai 3turn/5turn/parallel cassettes covering: Vec<ResponsesTool> parse, to_function_tool() normalization, ToolRegistry::build lookup, and full RequestPayload round-trip. DiscoveredTool, discover(), and the discovered param from normalize() all removed — no dead MCP code.

Also fixed two issues caught during review: ToolOutput missing #[derive(Debug)], and an MCP registry key mismatch (server_label was used as registry key but gateway_owned/client_owned look up by tool name — MCP entries now skip with a debug log until PR C adds real discovery).

Rebased on main after #79 and #81 — single clean commit, 264 tests pass.

@maralbahari

maralbahari commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

@ashwing thank you for addressing the issue.
after giving more thoughts on the ToolHandler design there some issues worth fixing now.

Trait shape: execute() on FunctionHandler seems meaningless.

ToolHandler has three methods: validate, normalize, and execute. The first two make sense for every tool type. But execute does not: the whole point of a function tool is that the gateway does not execute it. So FunctionHandler::execute() can never do anything valid, and the implementation reflects this by always returning an error.

The problem is that the trait forces every future tool handler to implement execute(), even ones that are also client-owned. The trait's shape implies "all tools can be executed by the gateway," which is false.

A cleaner split would be two traits:

pub trait ToolHandler {
    fn tool_type(&self) -> ToolType;
    fn validate(&self, param: &Value) -> Result<(), ToolError>;
    fn normalize(&self, param: &Value) -> Vec<FunctionTool>;
}

pub trait GatewayExecutor: ToolHandler {
    fn execute(
        &self,
        tool_name: &str,
        arguments: &str,
        config: &Value,
    ) -> Pin<Box<dyn Future<Output = Result<ToolOutput, ToolError>> + Send + '_>>;
}

FunctionHandler implements only ToolHandler. Future handlers for WebSearch, CodeInterpreter, etc. implement both. The type system then makes it impossible to accidentally call execute() on a client-owned tool, rather than relying on a runtime error to catch it.

ToolRegistry::gateway_owned() and client_owned() already do this split correctly at the routing layer. The trait should reflect the same boundary so the compiler enforces it, not a panic at runtime.

Comment thread crates/agentic-core/src/tool/normalize.rs Outdated
…n FunctionToolParam

Addresses Maral's review comments:

1. Split ToolHandler into two traits (design: every method must apply to
   all implementors):
   - ToolHandler: tool_type/validate/normalize — all tool types implement this
   - GatewayExecutor: ToolHandler + 'static + execute() — gateway-only tools
   FunctionHandler now implements only ToolHandler, making it a compile-time
   error to call execute() on a client-owned tool. Add _assert_gateway_executor_dyn_compatible
   test to lock in Arc<dyn GatewayExecutor> object safety.

2. Remove #[allow(unreachable_patterns)] + _ catch-all from normalize.rs
   and registry.rs — compiler now enforces exhaustiveness when new
   ResponsesTool variants are added (Maral's inline comment).

3. FunctionToolParam.name: String -> NonEmptyToolName — serde rejects
   empty names at deserialization time, eliminating three divergent
   empty-name checks (registry build, to_function_tool, FunctionHandler
   validate). FunctionHandler::normalize now returns vec![] + warn on
   invalid param instead of panicking with .expect().

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
@ashwing

ashwing commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author

@maralbahari The original design had execute() on ToolHandler as a forward-looking decision — PR B's HandlerRegistry was going to dispatch to Arc<dyn ToolHandler> uniformly, treating every tool through the same interface. The intent was to keep the dispatch loop simple: iterate calls, call execute() on whatever handler the registry returns, let the handler decide what to do. FunctionHandler::execute() returning an error was a guard for the case where routing went wrong, not the expected path.

That said, your point stands — gateway_owned/client_owned already do the routing split at the registry level, so the executor in PR B will never call execute() on a function tool. The split is the right call: ToolHandler for all tools, GatewayExecutor for the ones the gateway actually runs.

Done. Split into:

  • ToolHandler: tool_type, validate, normalize — every tool type implements this
  • GatewayExecutor: ToolHandler + 'static: execute() — gateway-owned tools only

FunctionHandler now implements only ToolHandler. Added a compile-time assertion _assert_gateway_executor_dyn_compatible to lock in Arc<dyn GatewayExecutor> object safety for PR B.

Also applied NonEmptyToolName to FunctionToolParam.name — serde now rejects empty names at deserialization, eliminating the three divergent empty-name checks across registry::build, to_function_tool, and FunctionHandler::validate.

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing Thanks. LGTM

@maralbahari maralbahari merged commit 8e1405a into vllm-project:main Jul 1, 2026
3 checks passed
ashwing added a commit to ashwing/agentic-api that referenced this pull request Jul 2, 2026
Implements the gateway-side agentic tool loop on top of the ToolRegistry
and GatewayExecutor traits landed in PR A (vllm-project#80):

- executor/dispatch.rs: LoopDecision enum (#[non_exhaustive]) +
  dispatch_tools() — classifies FunctionToolCall items via
  ToolRegistry::gateway_owned(), executes in parallel with 30s
  per-call timeout, maps failures to error-JSON FunctionCallOutput
  items (never aborts the loop on tool error).

- executor/agentic_loop.rs: execute_loop() — multi-turn orchestrator
  that clears all three persistence triggers before looping and
  restores original IDs on the final payload. Rejects stream=true
  (StreamTee is a future PR). Hard guard of 128 iterations, soft cap
  via max_iterations param (default: 10).

Client-owned function tools (ToolType::Function) return Done for now;
RequiresAction and ContinuePartial are deferred per staging agreement
in PR vllm-project#67 — LoopDecision is #[non_exhaustive] to make the addition safe.

MCP tool names are absent from the registry until PR C adds discovery;
any function_call for an MCP tool name is treated as client-owned.

244 tests pass; cargo clippy --workspace --all-targets -- -D warnings clean.

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants