From 9b66b332274a8d80cd621eb36c1e9050797e16fe Mon Sep 17 00:00:00 2001 From: Razvan Radulescu <43811028+h3xxit@users.noreply.github.com> Date: Tue, 9 Sep 2025 14:07:21 +0200 Subject: [PATCH] Update version 1.0 --- .../version-1.0/for-tool-providers.md | 17 +- versioned_docs/version-1.0/implementation.md | 1 - versioned_docs/version-1.0/index.md | 50 +- .../version-1.0/migration-v0.1-to-v1.0.md | 3 +- versioned_docs/version-1.0/protocols/http.md | 227 +++++++--- versioned_docs/version-1.0/protocols/index.md | 6 +- versioned_docs/version-1.0/protocols/mcp.md | 308 +++++++++---- versioned_docs/version-1.0/protocols/sse.md | 272 ++++++----- .../version-1.0/protocols/streamable-http.md | 222 +++++++-- versioned_docs/version-1.0/protocols/text.md | 426 +++++------------- versioned_docs/version-1.0/security.md | 25 - 11 files changed, 903 insertions(+), 654 deletions(-) diff --git a/versioned_docs/version-1.0/for-tool-providers.md b/versioned_docs/version-1.0/for-tool-providers.md index 87ff739..34457a1 100644 --- a/versioned_docs/version-1.0/for-tool-providers.md +++ b/versioned_docs/version-1.0/for-tool-providers.md @@ -45,7 +45,7 @@ As a tool provider, you'll create a **UTCP Manual** - a standardized description }, "tool_call_template": { "call_template_type": "http", - "url": "https://api.example.com/users/${user_id}", + "url": "https://api.example.com/users/{user_id}", "http_method": "GET" } } @@ -236,11 +236,7 @@ Use `${VARIABLE_NAME}` syntax for dynamic values: ```json { - "url": "https://api.example.com/users/${user_id}", - "body": { - "name": "${name}", - "email": "${email}" - } + "url": "https://api.example.com/users/{user_id}" } ``` @@ -270,8 +266,7 @@ Use `${VARIABLE_NAME}` syntax for dynamic values: "name": "get_data", "tool_call_template": { "call_template_type": "http", - "url": "${base_url}/data", - "timeout": "${timeout}" + "url": "${base_url}/data" } } ] @@ -327,7 +322,7 @@ The UTCP manual describes how to call your existing endpoints: }, "tool_call_template": { "call_template_type": "http", - "url": "https://api.example.com/users/${user_id}", + "url": "https://api.example.com/users/{user_id}", "http_method": "GET", "auth": { "auth_type": "api_key", @@ -550,9 +545,7 @@ Test your manual with UTCP clients: "call_template_type": "http", "url": "https://api.example.com/batch", "http_method": "POST", - "body": { - "items": "${items}" - } + "body_field": "items" } } ``` diff --git a/versioned_docs/version-1.0/implementation.md b/versioned_docs/version-1.0/implementation.md index c1aee3a..0eec31b 100644 --- a/versioned_docs/version-1.0/implementation.md +++ b/versioned_docs/version-1.0/implementation.md @@ -146,7 +146,6 @@ Call templates define how to invoke tools using specific protocols: } Tool arguments not used in the URL path or headers will be sent as query parameters for GET requests, or in the request body for POST/PUT/PATCH requests. The `body_field` specifies which tool argument contains the data for the request body. -} ``` #### CLI Call Template diff --git a/versioned_docs/version-1.0/index.md b/versioned_docs/version-1.0/index.md index 2355a3e..c9c4137 100644 --- a/versioned_docs/version-1.0/index.md +++ b/versioned_docs/version-1.0/index.md @@ -28,6 +28,10 @@ UTCP acts as a **"manual"** that tells agents how to call your tools directly: *"If a human can call your API, an AI agent should be able to call it too - with the same security and no additional infrastructure."* ::: +## OpenAPI Compatibility + +UTCP extends OpenAPI for AI agents while maintaining full backward compatibility. Where OpenAPI describes APIs for human developers, UTCP adds agent-focused enhancements: `tags` for categorization, `average_response_size` for resource planning, multi-protocol support (HTTP, CLI, gRPC, MCP), and direct execution instructions. Existing OpenAPI specs can be automatically converted to UTCP manuals without requiring API changes or additional infrastructure. + ## Quick Start (5 Minutes) ### 1. Install UTCP @@ -44,7 +48,12 @@ npm install @utcp/core @utcp/http ### 2. Expose Your First Tool -Add a discovery endpoint to your existing API: +**Option A: Discovery via existing OpenAPI spec** + +**Generate OpenAPI endpoint**: `GET http://api.github.com/openapi.json` + + +**Option B: Add a discovery endpoint to your existing API** **Add endpoint**: `GET /utcp` **Return your UTCP manual**: @@ -72,31 +81,55 @@ Add a discovery endpoint to your existing API: "var_name": "appid", "location": "query" } - } - }] } ``` ### 3. Call Your Tool -**Configure UTCP client**: +**Option A: Configure UTCP client**: ```json { "manual_call_templates": [{ "name": "weather_api", "call_template_type": "http", - "url": "http://localhost:8000/utcp", + "url": "http://localhost:8000/utcp", // Or http://api.github.com/openapi.json, the openapi spec gets converted automatically "http_method": "GET" }] } ``` +**Option B: Convert OpenAPI spec to UTCP manual manually** + +```python +async def convert_api(): + async with aiohttp.ClientSession() as session: + async with session.get("https://api.github.com/openapi.json") as response: + openapi_spec = await response.json() + + converter = OpenApiConverter(openapi_spec) + manual = converter.convert() + + print(f"Generated {len(manual.tools)} tools from GitHub API!") + return manual +``` + +Then save that to a text file and load it with the text configuration: +```json +{ + "manual_call_templates": [{ + "name": "github_manual", + "call_template_type": "text", + "file_path": "./github_manual.json", + }] +} +``` + + **Call the tool**: 1. Initialize UTCP client with configuration 2. Discover tools from weather API 3. Call `get_weather` tool with location parameter 4. Receive weather data response -``` **That's it!** Your tool is now discoverable and callable by any UTCP client. @@ -106,7 +139,7 @@ Add a discovery endpoint to your existing API: |---------|-------------| | **🚀 Zero Latency Overhead** | Direct tool calls, no proxy servers | | **🔒 Native Security** | Use your existing authentication and authorization | -| **🌐 Protocol Flexibility** | HTTP, WebSocket, CLI, GraphQL, and more | +| **🌐 Protocol Flexibility** | HTTP, MCP, CLI, GraphQL, and more | | **⚡ Easy Integration** | Add one endpoint, no infrastructure changes | | **📈 Scalable** | Leverage your existing scaling and monitoring | @@ -132,7 +165,6 @@ UTCP supports multiple communication protocols through plugins: | Protocol | Use Case | Plugin | Status | |----------|----------|--------|--------| | **[HTTP](./protocols/http.md)** | REST APIs, webhooks | `utcp-http` | ✅ Stable | -| **[WebSocket](./protocols/websocket.md)** | Real-time communication | `utcp-websocket` | ✅ Stable | | **[CLI](./protocols/cli.md)** | Command-line tools | `utcp-cli` | ✅ Stable | | **[Server-Sent Events](./protocols/streamable-http.md)** | Streaming data | `utcp-http` | ✅ Stable | | **[Text Files](./protocols/text.md)** | File reading | `utcp-text` | ✅ Stable | @@ -151,7 +183,7 @@ UTCP v1.0 features a modular, plugin-based architecture: - **[UTCP Client](./api/core/utcp/utcp_client.md)**: Tool discovery and execution engine ### Plugin System -- **Protocol Plugins**: HTTP, WebSocket, CLI, etc. +- **Protocol Plugins**: HTTP, MCP, CLI, etc. - **Custom Protocols**: Extend with your own communication methods - **Tool Repositories**: Pluggable storage for tool definitions - **Search Strategies**: Customizable tool discovery algorithms diff --git a/versioned_docs/version-1.0/migration-v0.1-to-v1.0.md b/versioned_docs/version-1.0/migration-v0.1-to-v1.0.md index eb50de7..38eb651 100644 --- a/versioned_docs/version-1.0/migration-v0.1-to-v1.0.md +++ b/versioned_docs/version-1.0/migration-v0.1-to-v1.0.md @@ -139,7 +139,7 @@ load_variables_from: ```json { "manual_version": "1.0.0", - "utcp_version": "1.0.1", + "utcp_version": "0.2.0", "info": { "title": "Weather API", "version": "1.0.0", @@ -174,7 +174,6 @@ load_variables_from: "var_name": "appid", "location": "query" } - } } ] } diff --git a/versioned_docs/version-1.0/protocols/http.md b/versioned_docs/version-1.0/protocols/http.md index 6340aeb..2c19f69 100644 --- a/versioned_docs/version-1.0/protocols/http.md +++ b/versioned_docs/version-1.0/protocols/http.md @@ -13,31 +13,29 @@ The HTTP protocol plugin enables UTCP to call tools via HTTP/HTTPS requests. Thi ```json { "call_template_type": "http", - "url": "https://api.example.com/endpoint", + "url": "https://api.example.com/users/{user_id}", "http_method": "GET|POST|PUT|DELETE|PATCH", + "content_type": "application/json", "headers": { - "Content-Type": "application/json", - "User-Agent": "UTCP-Client/1.0" + "X-Custom-Header": "static_value" }, - "query_params": { - "param1": "${variable1}", - "param2": "static_value" - }, - "body": { - "data": "${input_data}", - "timestamp": "${current_time}" - }, - "timeout": 30, - "verify_ssl": true, + "body_field": "body", + "header_fields": ["user_agent", "request_id"], "auth": { "auth_type": "api_key|basic|oauth2", "api_key": "${API_KEY}", - "var_name": "Authorization", - "location": "header" + "var_name": "Authorization", + "location": "header|query|cookie" } } ``` +### Field Descriptions + +For detailed field descriptions, examples, and implementation details, see: +- **[HttpCallTemplate API Reference](../api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md)** - Complete field documentation with examples +- **[HttpCommunicationProtocol API Reference](../api/plugins/communication_protocols/http/src/utcp_http/http_communication_protocol.md)** - Implementation details and method documentation + ## Supported HTTP Methods | Method | Use Case | Body Support | @@ -57,11 +55,16 @@ The HTTP protocol plugin enables UTCP to call tools via HTTP/HTTPS requests. Thi "auth_type": "api_key", "api_key": "${API_KEY}", "var_name": "X-API-Key", - "location": "header" + "location": "header|query|cookie" } } ``` +Supported locations: +- `"header"`: API key sent as HTTP header +- `"query"`: API key sent as query parameter +- `"cookie"`: API key sent as HTTP cookie + ### Basic Authentication ```json { @@ -86,37 +89,81 @@ The HTTP protocol plugin enables UTCP to call tools via HTTP/HTTPS requests. Thi } ``` -## Variable Substitution +## Parameter Handling + +The HTTP protocol handles tool arguments in a hierarchical manner: + +1. **URL Path Parameters**: Arguments matching `{param_name}` in the URL are substituted directly +2. **Body Field**: If `body_field` is specified, that argument becomes the request body +3. **Header Fields**: Arguments listed in `header_fields` become request headers +4. **Query Parameters**: All remaining arguments become query parameters -Variables in call templates are substituted with values from: -- Tool call arguments: `${argument_name}` -- Environment variables: `${ENV_VAR}` -- Configuration variables: `${config.variable}` +### URL Path Parameters +URL templates can include path parameters using `{parameter_name}` syntax: -Example: ```json { - "url": "https://api.example.com/users/${user_id}", - "headers": { - "Authorization": "Bearer ${ACCESS_TOKEN}" - }, - "query_params": { - "format": "${output_format}", - "limit": "${max_results}" + "url": "https://api.example.com/users/{user_id}/posts/{post_id}" +} +``` + +When calling a tool with arguments `{"user_id": "123", "post_id": "456", "limit": "10"}`, the URL becomes: +`https://api.example.com/users/123/posts/456?limit=10` + +### Body Field Mapping +Specify which tool argument should be sent as the request body: + +```json +{ + "body_field": "data", + "content_type": "application/json" +} +``` + +### Header Field Mapping +Map specific tool arguments to HTTP headers: + +```json +{ + "header_fields": ["user_agent", "request_id"] +} +``` + +Tool arguments `user_agent` and `request_id` will be sent as HTTP headers. + +### Variable Substitution in Authentication +Authentication fields support environment variable substitution: + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_KEY}", + "var_name": "Authorization", + "location": "header" } } ``` ## OpenAPI Integration -The HTTP protocol plugin can automatically generate UTCP manuals from OpenAPI/Swagger specifications: +The HTTP communication protocol automatically detects and converts OpenAPI/Swagger specifications to UTCP manuals: -1. **Automatic Discovery**: Point to an OpenAPI spec URL -2. **Schema Conversion**: Converts OpenAPI paths to UTCP tools -3. **Authentication Mapping**: Maps OpenAPI security schemes to UTCP auth -4. **Parameter Mapping**: Converts OpenAPI parameters to UTCP inputs +### Automatic Detection +When registering an HTTP provider, the protocol: +1. Fetches content from the specified URL +2. Checks if the response contains `utcp_version` and `tools` fields (UTCP manual) +3. If not, assumes it's an OpenAPI specification and converts it automatically -Example OpenAPI to UTCP conversion: +### Conversion Process +The `OpenApiConverter` handles: +- **Path Mapping**: OpenAPI paths become UTCP tool URLs with path parameters +- **Method Mapping**: HTTP methods are preserved +- **Parameter Mapping**: Path, query, header, and body parameters are mapped appropriately +- **Authentication**: OpenAPI security schemes are converted to UTCP auth configurations +- **Schema Validation**: OpenAPI schemas become UTCP input/output schemas + +### Example Conversion ```yaml # OpenAPI Specification paths: @@ -130,13 +177,13 @@ paths: type: string ``` -Becomes: +Becomes a UTCP tool: ```json { "name": "get_user", "tool_call_template": { "call_template_type": "http", - "url": "https://api.example.com/users/${id}", + "url": "https://api.example.com/users/{id}", "http_method": "GET" }, "inputs": { @@ -152,21 +199,31 @@ Becomes: ## Response Handling HTTP responses are processed based on: -- **Status Codes**: 2xx considered success, others as errors -- **Content-Type**: JSON, XML, text, and binary content support -- **Headers**: Response headers available in tool output -- **Error Mapping**: HTTP errors mapped to UTCP exceptions - -## Configuration Options - -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `timeout` | number | 30 | Request timeout in seconds | -| `verify_ssl` | boolean | true | Verify SSL certificates | -| `follow_redirects` | boolean | true | Follow HTTP redirects | -| `max_redirects` | number | 5 | Maximum redirect hops | -| `retry_count` | number | 0 | Number of retry attempts | -| `retry_delay` | number | 1 | Delay between retries (seconds) | +- **Status Codes**: 2xx considered success, 4xx/5xx raise exceptions +- **Content-Type**: + - `application/json`: Parsed as JSON object + - Other types: Returned as text string +- **Error Handling**: HTTP client errors are logged and re-raised +- **Timeouts**: 10 seconds for tool discovery, 30 seconds for tool execution + +## Security Features + +### HTTPS Enforcement +The HTTP protocol enforces secure connections by only allowing: +- HTTPS URLs (`https://`) +- Localhost URLs (`http://localhost` or `http://127.0.0.1`) + +Any other HTTP URLs will be rejected with a security error to prevent man-in-the-middle attacks. + +### OAuth2 Token Caching +OAuth2 tokens are automatically cached by `client_id` to avoid repeated authentication requests. The protocol supports both: +- **Body credentials**: Client ID/secret in request body +- **Header credentials**: Client ID/secret as Basic Auth header + +### Request Security +- URL path parameters are properly URL-encoded to prevent path injection +- All authentication credentials support environment variable substitution +- Cookies are supported for API key authentication when required ## Security Considerations @@ -188,10 +245,10 @@ HTTP responses are processed based on: - Implement rate limiting on the tool provider side ### Network Security -- Use HTTPS for all production communications -- Implement proper firewall rules -- Consider using VPNs or private networks for sensitive tools -- Monitor and log all HTTP requests for security analysis +- HTTPS is enforced by the protocol (except for localhost development) +- All credentials should use environment variable substitution (e.g., `${API_KEY}`) +- Path parameters are automatically URL-encoded to prevent injection attacks +- OAuth2 tokens are cached securely and not logged ## Error Handling @@ -210,10 +267,10 @@ Common HTTP errors and their meanings: ## Best Practices ### Performance -- Use connection pooling for multiple requests -- Implement appropriate timeouts -- Consider request/response compression -- Cache responses when appropriate +- Each request uses a fresh aiohttp ClientSession +- Tool discovery timeout: 10 seconds +- Tool execution timeout: 30 seconds +- OAuth2 tokens are cached to reduce authentication overhead ### Reliability - Implement retry logic with exponential backoff @@ -227,10 +284,58 @@ Common HTTP errors and their meanings: - Provide usage examples - Version your APIs and update call templates accordingly +## Complete Examples + +### Basic GET Request +```json +{ + "name": "get_user", + "call_template_type": "http", + "url": "https://api.example.com/users/{user_id}", + "http_method": "GET" +} +``` + +### POST with Authentication and Body +```json +{ + "name": "create_user", + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "content_type": "application/json", + "body_field": "user_data", + "header_fields": ["request_id"], + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_KEY}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +### OAuth2 Authentication +```json +{ + "name": "oauth_api", + "call_template_type": "http", + "url": "https://api.example.com/data", + "http_method": "GET", + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read write" + } +} +``` + ## Language-Specific Implementation For implementation details and examples in your programming language: -- **Multi-language**: [UTCP HTTP Protocol Examples](https://github.com/universal-tool-calling-protocol) - HTTP protocol examples across multiple languages +- **Python**: See `python-utcp/plugins/communication_protocols/http/` - **TypeScript**: [TypeScript HTTP Protocol Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/blob/main/docs/protocols/http.md) - **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) diff --git a/versioned_docs/version-1.0/protocols/index.md b/versioned_docs/version-1.0/protocols/index.md index 9a4f701..a6dfdc8 100644 --- a/versioned_docs/version-1.0/protocols/index.md +++ b/versioned_docs/version-1.0/protocols/index.md @@ -13,7 +13,6 @@ UTCP v1.0 features a modular, plugin-based architecture where different communic | Protocol | Plugin Package | Call Template | Use Cases | |----------|----------------|---------------|-----------| | **[HTTP](./http.md)** | `utcp-http` | `HttpCallTemplate` | REST APIs, webhooks, web services | -| **[WebSocket](./websocket.md)** | `utcp-websocket` | `WebSocketCallTemplate` | Real-time communication, streaming | | **[CLI](./cli.md)** | `utcp-cli` | `CliCallTemplate` | Command-line tools, scripts | | **[Server-Sent Events](./sse.md)** | `utcp-http` | `SseCallTemplate` | Event streaming, live updates | | **[Text Files](./text.md)** | `utcp-text` | `TextCallTemplate` | File reading, static content | @@ -43,10 +42,10 @@ Protocol plugins are available for different programming languages: ```bash # Example installation (Python) -pip install utcp-http utcp-cli utcp-websocket utcp-text utcp-mcp +pip install utcp-http utcp-cli utcp-text utcp-mcp # Example installation (Node.js) -npm install @utcp/http @utcp/cli @utcp/websocket @utcp/text @utcp/mcp +npm install @utcp/http @utcp/cli @utcp/text @utcp/mcp ``` For other languages, check the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) @@ -76,7 +75,6 @@ Example call template structure: Choose the right protocol plugin based on your needs: - **HTTP**: Most common for REST APIs and web services -- **WebSocket**: Real-time bidirectional communication - **CLI**: Wrapping existing command-line tools - **SSE**: Server-sent events for streaming data - **Text**: Reading configuration files or static content diff --git a/versioned_docs/version-1.0/protocols/mcp.md b/versioned_docs/version-1.0/protocols/mcp.md index 13da0f6..54f3d73 100644 --- a/versioned_docs/version-1.0/protocols/mcp.md +++ b/versioned_docs/version-1.0/protocols/mcp.md @@ -13,33 +13,61 @@ The MCP (Model Context Protocol) plugin provides interoperability between UTCP a ```json { "call_template_type": "mcp", - "server_config": { - "command": "node", - "args": ["mcp-server.js"], - "working_directory": "/app/mcp", - "env": { - "NODE_ENV": "production", - "LOG_LEVEL": "info" - }, - "timeout": 30 + "config": { + "mcpServers": { + "filesystem": { + "command": "node", + "args": ["mcp-server.js"], + "cwd": "/app/mcp", + "env": { + "NODE_ENV": "production", + "LOG_LEVEL": "info" + } + } + } }, - "connection_timeout": 10, - "request_timeout": 30 + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read:tools" + }, + "register_resources_as_tools": false } ``` +## Field Descriptions + +For detailed field specifications, examples, and validation rules, see: +- **[McpCallTemplate API Reference](../api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md)** - Complete field documentation with examples +- **[McpCommunicationProtocol API Reference](../api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_communication_protocol.md)** - Implementation details and method documentation + +### Key Fields + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `call_template_type` | string | Yes | - | Always "mcp" for MCP providers | +| `config` | object | Yes | - | Configuration object containing MCP server definitions | +| `auth` | object | No | null | Optional OAuth2 authentication for HTTP-based MCP servers | +| `register_resources_as_tools` | boolean | No | false | Whether to register MCP resources as callable tools | + ## Server Configuration -### Command-based Servers +### Command-based (stdio) Servers ```json { - "server_config": { - "command": "python", - "args": ["-m", "mcp_server", "--config", "config.json"], - "working_directory": "/app", - "env": { - "PYTHONPATH": "/app/lib", - "API_KEY": "${MCP_API_KEY}" + "config": { + "mcpServers": { + "my_server": { + "command": "python", + "args": ["-m", "mcp_server", "--config", "config.json"], + "cwd": "/app", + "env": { + "PYTHONPATH": "/app/lib", + "API_KEY": "${MCP_API_KEY}" + } + } } } } @@ -48,11 +76,41 @@ The MCP (Model Context Protocol) plugin provides interoperability between UTCP a ### HTTP-based Servers ```json { - "server_config": { - "transport": "http", - "url": "http://localhost:8080/mcp", - "headers": { - "Authorization": "Bearer ${MCP_TOKEN}" + "config": { + "mcpServers": { + "remote_server": { + "transport": "http", + "url": "https://mcp.example.com/api" + } + } + }, + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "mcp:access" + } +} +``` + +### Multiple Servers +```json +{ + "config": { + "mcpServers": { + "filesystem": { + "command": "node", + "args": ["filesystem-server.js"] + }, + "database": { + "command": "python", + "args": ["-m", "db_server"] + }, + "remote_api": { + "transport": "http", + "url": "https://api.example.com/mcp" + } } } } @@ -79,12 +137,13 @@ The MCP protocol plugin enables a gradual migration path from MCP to native UTCP ## Tool Discovery -MCP servers expose tools through the standard MCP protocol. The UTCP MCP plugin: +The MCP protocol implementation automatically discovers and maps tools: -1. **Connects** to the MCP server using stdio or HTTP transport -2. **Discovers** available tools via MCP's `tools/list` method -3. **Maps** MCP tool definitions to UTCP tool format -4. **Registers** tools in the UTCP client +1. **Session Management**: Creates persistent sessions with MCP servers using MCPClient +2. **Tool Discovery**: Lists available tools via MCP's `list_tools` method +3. **Tool Prefixing**: Adds server name prefix (e.g., `filesystem.read_file`) to ensure uniqueness +4. **Resource Support**: Optionally registers MCP resources as callable tools when `register_resources_as_tools` is true +5. **Tool Mapping**: Converts MCP tool schema to UTCP tool format automatically ## Request/Response Mapping @@ -105,9 +164,9 @@ MCP servers expose tools through the standard MCP protocol. The UTCP MCP plugin: // UTCP Tool (after mapping) { - "name": "read_file", + "name": "filesystem.read_file", "description": "Read contents of a file", - "inputs": { + "input_schema": { "type": "object", "properties": { "path": {"type": "string"} @@ -116,37 +175,65 @@ MCP servers expose tools through the standard MCP protocol. The UTCP MCP plugin: }, "tool_call_template": { "call_template_type": "mcp", - "server_config": {...} + "config": { + "mcpServers": {...} + } } } ``` ### Request Flow -1. UTCP client receives tool call -2. MCP plugin formats request as MCP `tools/call` -3. Request sent to MCP server -4. MCP server processes and responds -5. Response mapped back to UTCP format - -## Authentication and Security +1. UTCP client receives tool call with server-prefixed name (e.g., `filesystem.read_file`) +2. MCP plugin extracts server name and tool name +3. Gets or creates session with target MCP server +4. Calls MCP server's `call_tool` method +5. Processes response content (text, JSON, structured output) +6. Returns mapped result to UTCP client + +### Response Processing +The implementation intelligently processes MCP responses: +- **Structured output**: Returns `result.structured_output` if available +- **Text content**: Attempts JSON parsing, number parsing, or returns as string +- **List content**: Processes each item and returns as list or single item +- **Error handling**: Session-level errors trigger session restart + +## Authentication + +### OAuth2 Authentication (HTTP Servers) +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "mcp:read mcp:write" + } +} +``` -### Server Authentication +### Environment-based Authentication (stdio Servers) ```json { - "server_config": { - "command": "secure-mcp-server", - "env": { - "MCP_AUTH_TOKEN": "${MCP_SERVER_TOKEN}", - "MCP_CLIENT_ID": "${MCP_CLIENT_ID}" + "config": { + "mcpServers": { + "secure_server": { + "command": "secure-mcp-server", + "env": { + "MCP_AUTH_TOKEN": "${MCP_SERVER_TOKEN}", + "MCP_CLIENT_ID": "${MCP_CLIENT_ID}" + } + } } } } ``` -### Transport Security -- **stdio**: Inherits process security model -- **HTTP**: Use HTTPS and proper authentication headers -- **WebSocket**: Use WSS and authentication tokens +### Security Features +- **OAuth2 token caching**: Tokens cached by client_id to avoid repeated requests +- **Session management**: Persistent sessions with automatic error recovery +- **Environment variables**: Use `${VAR_NAME}` syntax for sensitive credentials +- **Transport security**: stdio inherits process security, HTTP supports OAuth2 ## Error Handling @@ -184,53 +271,70 @@ MCP errors are mapped to UTCP exceptions: ## Limitations -### MCP Feature Support -Not all MCP features are supported through UTCP: -- **Resources**: Not directly mapped to UTCP tools +### Current Limitations - **Prompts**: Not supported in UTCP model - **Sampling**: Not applicable to tool calling +- **Streaming**: MCP streaming calls return single result (no streaming support) -### Protocol Differences -- MCP's bidirectional communication vs UTCP's request/response -- MCP's resource model vs UTCP's tool-only model -- Different authentication mechanisms +### MCP Feature Support +Full support for core MCP features: +- **Tools**: Complete tool discovery and execution support +- **Resources**: Optional support via `register_resources_as_tools` flag +- **Authentication**: OAuth2 support for HTTP-based servers +- **Session management**: Persistent sessions with automatic recovery +- **Multiple servers**: Single provider can manage multiple MCP servers + +### Protocol Mapping +- **Tool naming**: Server-prefixed names ensure uniqueness across multiple servers +- **Response processing**: Intelligent parsing of MCP response formats +- **Error handling**: Session-level vs protocol-level error distinction +- **Resource tools**: Resources exposed as callable tools when enabled ## Configuration Examples ### Development Setup ```json { - "manual_call_templates": [{ - "name": "dev_mcp", - "call_template_type": "mcp", - "server_config": { - "command": "node", - "args": ["dev-server.js"], - "env": {"NODE_ENV": "development"} - }, - "connection_timeout": 5, - "request_timeout": 10 - }] + "name": "dev_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "filesystem": { + "command": "node", + "args": ["dev-server.js"], + "env": {"NODE_ENV": "development"} + }, + "database": { + "command": "python", + "args": ["-m", "db_server", "--dev"] + } + } + }, + "register_resources_as_tools": true } ``` ### Production Setup ```json { - "manual_call_templates": [{ - "name": "prod_mcp", - "call_template_type": "mcp", - "server_config": { - "transport": "http", - "url": "https://mcp.example.com/api", - "headers": { - "Authorization": "Bearer ${MCP_PROD_TOKEN}", - "X-Client-Version": "1.0.0" + "name": "prod_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "api_server": { + "transport": "http", + "url": "https://mcp.example.com/api" } - }, - "connection_timeout": 30, - "request_timeout": 60 - }] + } + }, + "auth": { + "auth_type": "oauth2", + "client_id": "${MCP_CLIENT_ID}", + "client_secret": "${MCP_CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "mcp:access" + }, + "register_resources_as_tools": false } ``` @@ -257,10 +361,44 @@ Not all MCP features are supported through UTCP: - Monitor for suspicious activity - Keep MCP servers updated -## Language-Specific Implementation +## Implementation Notes + +The MCP protocol implementation provides: + +- **Session persistence**: Reuses MCP sessions for better performance +- **Automatic recovery**: Handles session failures with automatic retry +- **Multi-server support**: Single provider manages multiple MCP servers +- **Resource integration**: Optional resource-to-tool mapping +- **OAuth2 support**: Full OAuth2 authentication for HTTP servers +- **Intelligent response processing**: Handles various MCP response formats + +### Usage Example +```python +import asyncio +from utcp_client import UtcpClient + +async def main(): + client = UtcpClient() + + # Register MCP provider with multiple servers + await client.register_tool_provider(mcp_manual) + + # Call tools with server-prefixed names + result = await client.call_tool("filesystem.read_file", {"path": "/data/file.txt"}) + + # Access resources as tools (if enabled) + resource_data = await client.call_tool("filesystem.resource_config", {}) + + await client.close() + +if __name__ == "__main__": + asyncio.run(main()) +``` + +## Related Protocols -For implementation details and examples in your programming language: +- **[HTTP](./http.md)** - For native HTTP-based tool implementations +- **[Server-Sent Events (SSE)](./sse.md)** - For real-time streaming tools +- **TCP/UDP** - For custom protocol implementations -- **Multi-language**: [UTCP MCP Protocol Examples](https://github.com/universal-tool-calling-protocol) - MCP protocol examples across multiple languages -- **TypeScript**: [TypeScript MCP Protocol Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/blob/main/docs/protocols/mcp.md) -- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) +For complete implementation details, see the [MCP Communication Protocol API Reference](../api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_communication_protocol.md). diff --git a/versioned_docs/version-1.0/protocols/sse.md b/versioned_docs/version-1.0/protocols/sse.md index 93618e9..4567605 100644 --- a/versioned_docs/version-1.0/protocols/sse.md +++ b/versioned_docs/version-1.0/protocols/sse.md @@ -25,56 +25,85 @@ npm install @utcp/http ```json { "call_template_type": "sse", - "url": "https://api.example.com/events", + "url": "https://api.example.com/events/{stream_id}", + "event_type": "data_update", + "reconnect": true, + "retry_timeout": 30000, "headers": { - "Authorization": "Bearer ${API_TOKEN}", - "Accept": "text/event-stream" + "X-Custom-Header": "static_value" }, - "timeout": 60, - "max_events": 10, - "event_filter": { - "type": "data_update" + "body_field": "payload", + "header_fields": ["user_id", "session_token"], + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_TOKEN}", + "var_name": "Authorization", + "location": "header" } } ``` -## Configuration Options +## Field Descriptions -The Server-Sent Events (SSE) call template enables real-time streaming data from HTTP endpoints. For complete field specifications and validation rules, see the [SSE Call Template API Reference](../api/plugins/communication_protocols/http/src/utcp_http/sse_call_template.md). -| `reconnect` | boolean | Auto-reconnect on connection loss (default: true) | -| `reconnect_delay` | number | Delay between reconnection attempts (default: 3) | +For detailed field specifications, examples, and validation rules, see: +- **[SseCallTemplate API Reference](../api/plugins/communication_protocols/http/src/utcp_http/sse_call_template.md)** - Complete field documentation with examples +- **[SseCommunicationProtocol API Reference](../api/plugins/communication_protocols/http/src/utcp_http/sse_communication_protocol.md)** - Implementation details and method documentation -## Authentication +### Key Fields -SSE uses standard HTTP authentication methods: +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `call_template_type` | string | Yes | Always "sse" for SSE providers | +| `url` | string | Yes | SSE endpoint URL with optional path parameters like `{stream_id}` | +| `event_type` | string | No | Filter for specific event types (default: all events) | +| `reconnect` | boolean | No | Auto-reconnect on connection loss (default: true) | +| `retry_timeout` | number | No | Retry timeout in milliseconds (default: 30000) | +| `headers` | object | No | Static headers for the initial connection | +| `body_field` | string | No | Tool argument name to map to request body | +| `header_fields` | array | No | Tool argument names to map to request headers | +| `auth` | object | No | Authentication configuration | + +## Authentication -### Bearer Token +SSE supports the same authentication methods as HTTP: +### API Key Authentication ```json { - "headers": { - "Authorization": "Bearer ${ACCESS_TOKEN}" + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${ACCESS_TOKEN}", + "var_name": "Authorization", + "location": "header" } } ``` -### API Key +Supported locations: +- `"header"`: API key sent as HTTP header +- `"query"`: API key sent as query parameter +- `"cookie"`: API key sent as HTTP cookie +### Basic Authentication ```json { - "headers": { - "X-API-Key": "${API_KEY}" + "auth": { + "auth_type": "basic", + "username": "${USERNAME}", + "password": "${PASSWORD}" } } ``` -### Query Parameter Auth - +### OAuth2 Authentication ```json { - "query_params": { - "token": "${API_TOKEN}", - "user_id": "${USER_ID}" + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read write" } } ``` @@ -96,14 +125,16 @@ SSE uses standard HTTP authentication methods: }, "tool_call_template": { "call_template_type": "sse", - "url": "https://api.example.com/notifications/stream", - "query_params": { - "user_id": "${user_id}" - }, - "headers": { - "Authorization": "Bearer ${ACCESS_TOKEN}" - }, - "timeout": 300 + "url": "https://api.example.com/notifications/stream/{user_id}", + "event_type": "notification", + "reconnect": true, + "retry_timeout": 30000, + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${ACCESS_TOKEN}", + "var_name": "Authorization", + "location": "header" + } } } ``` @@ -114,11 +145,9 @@ SSE uses standard HTTP authentication methods: { "call_template_type": "sse", "url": "https://api.example.com/events", - "event_filter": { - "type": "order_update", - "status": ["completed", "cancelled"] - }, - "max_events": 5 + "event_type": "order_update", + "reconnect": true, + "retry_timeout": 15000 } ``` @@ -133,27 +162,21 @@ SSE uses standard HTTP authentication methods: "inputs": { "type": "object", "properties": { - "symbols": { - "type": "array", - "items": {"type": "string"} - }, - "duration": {"type": "number", "default": 60} + "symbol": {"type": "string"} }, - "required": ["symbols"] + "required": ["symbol"] }, "tool_call_template": { "call_template_type": "sse", - "url": "https://api.stocks.com/stream", - "query_params": { - "symbols": "${symbols}", - "format": "json" - }, - "headers": { - "Authorization": "Bearer ${STOCK_API_KEY}" - }, - "timeout": "${duration}", - "event_filter": { - "type": "price_update" + "url": "https://api.stocks.com/stream/{symbol}", + "event_type": "price_update", + "reconnect": true, + "retry_timeout": 30000, + "auth": { + "auth_type": "api_key", + "api_key": "${STOCK_API_KEY}", + "var_name": "Authorization", + "location": "header" } } } @@ -169,22 +192,24 @@ SSE uses standard HTTP authentication methods: "type": "object", "properties": { "service": {"type": "string"}, - "level": {"type": "string", "enum": ["error", "warn", "info", "debug"]} + "level": {"type": "string"} }, "required": ["service"] }, "tool_call_template": { "call_template_type": "sse", - "url": "https://logs.example.com/stream", - "query_params": { - "service": "${service}", - "level": "${level}" - }, - "headers": { - "X-API-Key": "${LOG_API_KEY}" - }, - "timeout": 600, - "max_events": 100 + "url": "https://logs.example.com/stream/{service}", + "event_type": "log_entry", + "reconnect": true, + "retry_timeout": 45000, + "body_field": "filter_config", + "header_fields": ["level"], + "auth": { + "auth_type": "api_key", + "api_key": "${LOG_API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } } } ``` @@ -198,27 +223,25 @@ SSE uses standard HTTP authentication methods: "inputs": { "type": "object", "properties": { - "metrics": { - "type": "array", - "items": {"type": "string"} - }, - "interval": {"type": "number", "default": 5} + "server_id": {"type": "string"}, + "metrics_config": {"type": "object"} }, - "required": ["metrics"] + "required": ["server_id"] }, "tool_call_template": { "call_template_type": "sse", - "url": "https://monitoring.example.com/metrics/stream", - "query_params": { - "metrics": "${metrics}", - "interval": "${interval}" - }, - "headers": { - "Authorization": "Bearer ${MONITORING_TOKEN}" - }, - "timeout": 300, + "url": "https://monitoring.example.com/metrics/stream/{server_id}", + "event_type": "metric_update", "reconnect": true, - "reconnect_delay": 5 + "retry_timeout": 20000, + "body_field": "metrics_config", + "auth": { + "auth_type": "oauth2", + "client_id": "${MONITORING_CLIENT_ID}", + "client_secret": "${MONITORING_CLIENT_SECRET}", + "token_url": "https://auth.monitoring.com/token", + "scope": "metrics:read" + } } } ``` @@ -318,47 +341,86 @@ data: {"message": "Simple data without event type"} ## Advanced Features -### Custom Event Parsing +### Dynamic Parameter Substitution +- **URL path parameters**: Use `{parameter_name}` syntax in URLs +- **Body field mapping**: Map tool arguments to request body via `body_field` +- **Header field mapping**: Map tool arguments to headers via `header_fields` ```json { "call_template_type": "sse", - "url": "https://api.example.com/events", - "event_parser": { - "format": "json", - "extract_fields": ["timestamp", "level", "message"] - } + "url": "https://api.example.com/events/{stream_id}", + "event_type": "data_update", + "body_field": "filter_config", + "header_fields": ["user_context", "session_id"] } ``` -### Event Aggregation +### OAuth2 Token Management +- **Automatic token caching**: Tokens cached by client_id +- **Token refresh**: Automatic token refresh on expiration +- **Client credentials flow**: Supports OAuth2 client credentials grant ```json { - "call_template_type": "sse", - "url": "https://api.example.com/metrics", - "aggregation": { - "window": 10, - "function": "average", - "field": "value" + "auth": { + "auth_type": "oauth2", + "client_id": "${OAUTH_CLIENT_ID}", + "client_secret": "${OAUTH_CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "stream:read" } } ``` -### Conditional Termination +### Multiple Authentication Locations +- **Header**: Standard Authorization header or custom headers +- **Query**: API key as URL query parameter +- **Cookie**: API key sent as HTTP cookie ```json { - "call_template_type": "sse", - "url": "https://api.example.com/events", - "termination_condition": { - "event_type": "complete", - "data_field": "status", - "value": "finished" + "auth": { + "auth_type": "api_key", + "api_key": "${API_TOKEN}", + "var_name": "access_token", + "location": "cookie" } } ``` +## Implementation Notes + +The SSE protocol implementation provides: + +- **Async streaming**: Real-time event processing with async generators +- **Automatic reconnection**: Configurable via `reconnect` and `retry_timeout` fields +- **Event filtering**: Client-side filtering by `event_type` +- **Authentication caching**: OAuth2 tokens cached by client_id +- **Security enforcement**: HTTPS or localhost connections only +- **Error handling**: Graceful handling of connection failures and retries + +### Usage Example +```python +import asyncio +from utcp_client import UtcpClient + +async def main(): + client = UtcpClient() + + # Register SSE provider + await client.register_tool_provider(sse_manual) + + # Stream events with automatic filtering and reconnection + async for event in client.call_tool_streaming("stream_notifications", {"user_id": "123"}): + print(f"Event: {event}") + + await client.close() + +if __name__ == "__main__": + asyncio.run(main()) +``` + ## Common Use Cases - **Real-time Dashboards**: Live metrics, status updates @@ -381,6 +443,8 @@ data: {"message": "Simple data without event type"} ## Related Protocols -- [HTTP](./http.md) - For request/response patterns -- [WebSocket](./websocket.md) - For bidirectional communication -- [Streamable HTTP](./streamable-http.md) - For chunked HTTP responses +- **[HTTP](./http.md)** - For standard request/response patterns +- **WebSocket** - For bidirectional real-time communication +- **TCP/UDP** - For custom protocol implementations + +For complete implementation details, see the [SSE Communication Protocol API Reference](../api/plugins/communication_protocols/http/src/utcp_http/sse_communication_protocol.md). diff --git a/versioned_docs/version-1.0/protocols/streamable-http.md b/versioned_docs/version-1.0/protocols/streamable-http.md index 3ed9e6e..1a2341d 100644 --- a/versioned_docs/version-1.0/protocols/streamable-http.md +++ b/versioned_docs/version-1.0/protocols/streamable-http.md @@ -13,8 +13,11 @@ The Streamable HTTP protocol plugin (`utcp-http`) enables UTCP to handle large H ```json { "call_template_type": "streamable_http", - "url": "https://api.example.com/download/large-file", + "url": "https://api.example.com/download/{file_id}", "http_method": "GET", + "content_type": "application/octet-stream", + "chunk_size": 4096, + "timeout": 60000, "headers": { "Accept": "application/octet-stream" }, @@ -24,28 +27,31 @@ The Streamable HTTP protocol plugin (`utcp-http`) enables UTCP to handle large H "var_name": "Authorization", "location": "header" }, - "chunk_size": 8192, - "timeout": 300000, "body_field": "request_data", "header_fields": ["custom_header_arg"] } ``` -## Configuration Options +## Field Descriptions -The Streamable HTTP call template provides a way to configure streaming from HTTP endpoints. +For detailed field specifications, examples, and validation rules, see: +- **[StreamableHttpCallTemplate API Reference](../api/plugins/communication_protocols/http/src/utcp_http/streamable_http_call_template.md)** - Complete field documentation with examples +- **[StreamableHttpCommunicationProtocol API Reference](../api/plugins/communication_protocols/http/src/utcp_http/streamable_http_communication_protocol.md)** - Implementation details and method documentation -| Option | Type | Default | Description | -|---|---|---|---| -| `url` | string | **Required** | The streaming HTTP endpoint URL. Supports path parameters like `/users/{user_id}`. | -| `http_method` | string | `GET` | The HTTP method to use. Supported methods are `GET` and `POST`. | -| `content_type` | string | `application/octet-stream` | The `Content-Type` header to set for the request, especially when `body_field` is used. | -| `chunk_size` | integer | `4096` | The size of each data chunk in bytes to read from the stream. | -| `timeout` | integer | `60000` | Request timeout in milliseconds. | -| `headers` | object | `null` | Optional static headers to include in every request. | -| `auth` | object | `null` | Optional authentication configuration. See [HTTP Authentication](./http.md#authentication-methods). | -| `body_field` | string | `null` | The name of a single tool argument to be sent as the HTTP request body. | -| `header_fields` | array | `null` | A list of tool argument names to be sent as request headers. | +### Key Fields + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `call_template_type` | string | Yes | - | Always "streamable_http" for streaming HTTP providers | +| `url` | string | Yes | - | HTTP endpoint URL with optional path parameters like `{file_id}` | +| `http_method` | string | No | "GET" | HTTP method to use ("GET" or "POST") | +| `content_type` | string | No | "application/octet-stream" | Content-Type header for requests | +| `chunk_size` | number | No | 4096 | Size of each data chunk in bytes | +| `timeout` | number | No | 60000 | Request timeout in milliseconds | +| `headers` | object | No | null | Static headers to include in requests | +| `auth` | object | No | null | Authentication configuration | +| `body_field` | string | No | null | Tool argument name to map to request body | +| `header_fields` | array | No | null | Tool argument names to map to request headers | ## Response Handling @@ -58,32 +64,90 @@ The protocol processes the incoming stream based on the `Content-Type` header of ## Authentication -Streamable HTTP supports the same authentication methods as the standard HTTP protocol, including API Key, Basic Auth, and OAuth2. The configuration is identical. +Streamable HTTP supports the same authentication methods as HTTP: + +### API Key Authentication +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${ACCESS_TOKEN}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +Supported locations: +- `"header"`: API key sent as HTTP header +- `"query"`: API key sent as query parameter +- `"cookie"`: API key sent as HTTP cookie + +### Basic Authentication +```json +{ + "auth": { + "auth_type": "basic", + "username": "${USERNAME}", + "password": "${PASSWORD}" + } +} +``` + +### OAuth2 Authentication +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read write" + } +} +``` + +## Parameter Handling -For more details, see the [HTTP Authentication Methods](./http.md#authentication-methods) documentation. +Streamable HTTP processes tool arguments through a hierarchy: -## Variable Substitution +1. **URL path parameters**: Substituted using `{parameter_name}` syntax +2. **Body field**: Single argument mapped to request body via `body_field` +3. **Header fields**: Arguments mapped to headers via `header_fields` +4. **Query parameters**: Remaining arguments sent as query parameters -Path parameters, query parameters, headers, and authentication fields all support variable substitution from tool arguments and environment variables, following the same syntax as the standard HTTP protocol. +### URL Path Parameters +Parameters in URLs are substituted from tool arguments: +```json +{ + "url": "https://api.example.com/files/{file_id}/download/{format}" +} +``` +Tool arguments `file_id` and `format` are substituted into the URL path. -Example: +### Variable Substitution +Authentication and header values support environment variable substitution: ```json { - "url": "https://api.example.com/files/{file_id}/download", "headers": { "Authorization": "Bearer ${ACCESS_TOKEN}" } } ``` -Here, `{file_id}` is substituted from a tool argument, and `${ACCESS_TOKEN}` is substituted from an environment or configuration variable. - -For more details, see the [HTTP Variable Substitution](./http.md#variable-substitution) documentation. ## Security Considerations -- **SSL/TLS**: It is strongly recommended to use `https://` endpoints to protect data in transit. The implementation enforces HTTPS or localhost connections by default. -- **Authentication**: Never hardcode credentials. Use variable substitution to inject secrets from a secure source (e.g., environment variables). -- **Input Sanitization**: Ensure that any arguments used in URL path parameters or query strings are properly validated and sanitized to prevent injection attacks. +### Connection Security +- **HTTPS enforcement**: Only HTTPS URLs or localhost connections are allowed +- **Certificate validation**: SSL certificates are validated by default +- **Secure token handling**: OAuth2 tokens are cached securely + +### Authentication Security +- **Environment variables**: Use `${VAR_NAME}` syntax for sensitive credentials +- **Token caching**: OAuth2 tokens are cached by client_id to avoid repeated requests +- **Authentication methods**: Support for API key, Basic auth, and OAuth2 +- **Location flexibility**: API keys can be sent in headers, query params, or cookies +- **URL encoding**: Path parameters are properly URL-encoded to prevent injection ## Error Handling @@ -99,7 +163,105 @@ Errors are handled similarly to the standard HTTP protocol: Connection errors, timeouts, and other network issues will also be raised as exceptions. +## Usage Examples + +### Large File Download +```json +{ + "name": "download_dataset", + "description": "Download large dataset files", + "inputs": { + "type": "object", + "properties": { + "dataset_id": {"type": "string"}, + "format": {"type": "string", "enum": ["csv", "json", "parquet"]} + }, + "required": ["dataset_id"] + }, + "tool_call_template": { + "call_template_type": "streamable_http", + "url": "https://data.example.com/datasets/{dataset_id}/download", + "http_method": "GET", + "chunk_size": 8192, + "timeout": 300000, + "header_fields": ["format"], + "auth": { + "auth_type": "api_key", + "api_key": "${DATA_API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } + } +} +``` + +### Streaming JSON Data +```json +{ + "name": "export_records", + "description": "Export large record sets as streaming NDJSON", + "inputs": { + "type": "object", + "properties": { + "table_name": {"type": "string"}, + "filters": {"type": "object"} + }, + "required": ["table_name"] + }, + "tool_call_template": { + "call_template_type": "streamable_http", + "url": "https://api.example.com/export/{table_name}", + "http_method": "POST", + "content_type": "application/json", + "chunk_size": 4096, + "body_field": "filters", + "auth": { + "auth_type": "oauth2", + "client_id": "${OAUTH_CLIENT_ID}", + "client_secret": "${OAUTH_CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "data:export" + } + } +} +``` + +## Implementation Notes + +The Streamable HTTP protocol implementation provides: + +- **Chunked streaming**: Processes responses in configurable chunk sizes +- **Content-type awareness**: Different handling for JSON, NDJSON, and binary content +- **Authentication caching**: OAuth2 tokens cached by client_id +- **Security enforcement**: HTTPS or localhost connections only +- **Error handling**: Graceful handling of connection failures and timeouts +- **Resource management**: Proper cleanup of HTTP connections and sessions + +### Usage Example +```python +import asyncio +from utcp_client import UtcpClient + +async def main(): + client = UtcpClient() + + # Register streamable HTTP provider + await client.register_tool_provider(streamable_http_manual) + + # Stream large dataset + async for chunk in client.call_tool_streaming("download_dataset", {"dataset_id": "large_dataset_123"}): + process_chunk(chunk) # Process each chunk as it arrives + + await client.close() + +if __name__ == "__main__": + asyncio.run(main()) +``` + ## Related Protocols -- [HTTP](./http.md) - For standard request/response interactions. -- [Server-Sent Events (SSE)](./sse.md) - For unidirectional, real-time event streams from server to client. \ No newline at end of file +- **[HTTP](./http.md)** - For standard request/response interactions +- **[Server-Sent Events (SSE)](./sse.md)** - For real-time event streams from server to client +- **TCP/UDP** - For custom protocol implementations + +For complete implementation details, see the [Streamable HTTP Communication Protocol API Reference](../api/plugins/communication_protocols/http/src/utcp_http/streamable_http_communication_protocol.md). \ No newline at end of file diff --git a/versioned_docs/version-1.0/protocols/text.md b/versioned_docs/version-1.0/protocols/text.md index dcca388..885777b 100644 --- a/versioned_docs/version-1.0/protocols/text.md +++ b/versioned_docs/version-1.0/protocols/text.md @@ -6,16 +6,13 @@ sidebar_position: 5 # Text Protocol -The Text protocol plugin (`utcp-text`) enables UTCP to read and process text files from local filesystem or remote URLs. This is useful for tools that need to access documentation, configuration files, logs, or any text-based data. +The Text protocol plugin (`utcp-text`) enables UTCP to read UTCP manuals and tool definitions from local JSON/YAML files. This protocol is designed for static tool configurations or environments where manuals are distributed as files. ## Installation ```bash -# Example installation (Python) +# Python installation pip install utcp-text - -# Example installation (Node.js) -npm install @utcp/text ``` ## Call Template Structure @@ -23,415 +20,202 @@ npm install @utcp/text ```json { "call_template_type": "text", - "file_path": "/path/to/file.txt", - "encoding": "utf-8", - "max_size": 1048576, - "line_range": { - "start": 1, - "end": 100 - } + "file_path": "/path/to/manual.json" } ``` ## Configuration Options -The Text call template enables reading and processing text files from local filesystem or URLs. For complete field specifications and validation rules, see the [Text Call Template API Reference](../api/plugins/communication_protocols/text/src/utcp_text/text_call_template.md). +The Text call template has a simple structure for reading UTCP manuals from files. For complete field specifications and validation rules, see the [Text Call Template API Reference](../api/plugins/communication_protocols/text/src/utcp_text/text_call_template.md). -## File Sources +### Required Fields -### Local Files +- **`call_template_type`**: Must be "text" +- **`file_path`**: Path to the file containing UTCP manual or tool definitions +- **`auth`**: Always `None` (text call templates don't support authentication) -```json -{ - "call_template_type": "text", - "file_path": "/var/log/application.log", - "encoding": "utf-8" -} -``` +## Supported File Types -### Remote URLs +### UTCP Manual Files ```json { "call_template_type": "text", - "file_path": "https://example.com/config.txt", - "max_size": 512000 + "file_path": "/path/to/utcp_manual.json" } ``` -### Variable Substitution +### OpenAPI Specifications + +The protocol automatically detects and converts OpenAPI specs to UTCP manuals: ```json { "call_template_type": "text", - "file_path": "/data/${filename}", - "encoding": "${file_encoding}" + "file_path": "/path/to/openapi.yaml" } ``` -## Examples +### Path Resolution -### Read Configuration File +Relative paths are resolved against the UTCP client's root directory: ```json { - "name": "read_config", - "description": "Read application configuration file", - "inputs": { - "type": "object", - "properties": { - "config_name": {"type": "string"} - }, - "required": ["config_name"] - }, - "tool_call_template": { - "call_template_type": "text", - "file_path": "/etc/app/${config_name}.conf", - "encoding": "utf-8", - "max_size": 65536 - } + "call_template_type": "text", + "file_path": "manuals/my_tools.json" // Resolved against client root_dir } ``` -### Read Log File with Line Range +## Usage Examples + +### Manual Registration + +#### UTCP Manual File ```json { - "name": "read_recent_logs", - "description": "Read recent log entries", - "inputs": { - "type": "object", - "properties": { - "log_file": {"type": "string"}, - "lines": {"type": "number", "default": 100} - }, - "required": ["log_file"] - }, - "tool_call_template": { + "name": "local_tools_manual", + "call_template": { "call_template_type": "text", - "file_path": "/var/log/${log_file}", - "line_range": { - "start": -${lines}, - "end": -1 - } + "file_path": "/path/to/my_tools.json" } } ``` -### Read Remote Documentation +#### OpenAPI Specification ```json { - "name": "fetch_documentation", - "description": "Fetch documentation from remote URL", - "inputs": { - "type": "object", - "properties": { - "doc_url": {"type": "string"}, - "section": {"type": "string"} - }, - "required": ["doc_url"] - }, - "tool_call_template": { + "name": "api_tools_manual", + "call_template": { "call_template_type": "text", - "file_path": "${doc_url}", - "pattern": "(?s)## ${section}.*?(?=## |$)", - "max_size": 2097152 + "file_path": "/path/to/openapi.yaml" } } ``` -### Search in File +### Tool Execution + +When tools are called, the Text protocol returns the content of the configured file: ```json { - "name": "search_in_file", - "description": "Search for pattern in text file", + "name": "read_file", + "description": "Read content from a file", "inputs": { "type": "object", - "properties": { - "file_path": {"type": "string"}, - "search_pattern": {"type": "string"} - }, - "required": ["file_path", "search_pattern"] + "properties": {}, + "required": [] }, "tool_call_template": { "call_template_type": "text", - "file_path": "${file_path}", - "pattern": "${search_pattern}", - "transform": "strip" - } -} -``` - -## Line Range Options - -### Absolute Line Numbers - -```json -{ - "line_range": { - "start": 10, - "end": 50 - } -} -``` - -### Relative to End (Tail) - -```json -{ - "line_range": { - "start": -100, - "end": -1 + "file_path": "/path/to/data.txt" } } ``` -### From Start (Head) - -```json -{ - "line_range": { - "start": 1, - "end": 100 - } -} -``` +## Protocol Behavior -## Pattern Matching - -### Simple Text Search - -```json -{ - "pattern": "ERROR" -} -``` - -### Regex Pattern - -```json -{ - "pattern": "\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} ERROR.*" -} -``` +### Manual Registration -### Multi-line Pattern +When registering a manual with `register_manual()`, the protocol: -```json -{ - "pattern": "(?s)START.*?END" -} -``` +1. **Reads the file** from the specified `file_path` +2. **Detects file format** (JSON/YAML) +3. **Identifies content type**: + - UTCP Manual: Validates and returns directly + - OpenAPI Spec: Converts to UTCP manual using OpenApiConverter +4. **Returns result** with loaded tools -## Content Transformations +### Tool Execution -### Case Transformations +When calling a tool with `call_tool()`, the protocol: -```json -{ - "transform": "upper" // Convert to uppercase -} -``` +1. **Reads file content** from the `file_path` in the tool's call template +2. **Returns raw content** as a string -```json -{ - "transform": "lower" // Convert to lowercase -} -``` +### Streaming Support -### Whitespace Handling +The `call_tool_streaming()` method yields the entire file content as a single chunk. -```json -{ - "transform": "strip" // Remove leading/trailing whitespace -} -``` +## File Format Support -### Custom Transformations +### JSON Files ```json +// /path/to/manual.json { - "transform": "normalize_whitespace" // Normalize all whitespace + "version": "0.2.0", + "tools": [ + { + "name": "example_tool", + "description": "An example tool", + "inputs": {"type": "object", "properties": {}} + } + ] } ``` -## Response Format - -### Successful Read +### YAML Files -```json -{ - "content": "File content here...", - "metadata": { - "file_path": "/path/to/file.txt", - "size": 1024, - "lines": 25, - "encoding": "utf-8", - "last_modified": "2024-01-15T10:30:00Z" - } -} -``` - -### Filtered Content - -```json -{ - "content": "Matching lines...", - "metadata": { - "file_path": "/path/to/file.txt", - "total_lines": 1000, - "matched_lines": 5, - "pattern": "ERROR", - "line_range": {"start": 1, "end": 100} - } -} +```yaml +# /path/to/manual.yaml +version: "0.2.0" +tools: + - name: example_tool + description: An example tool + inputs: + type: object + properties: {} ``` ## Error Handling -| Error Type | Description | Handling | -|------------|-------------|----------| -| File Not Found | File doesn't exist | Raise `FileNotFoundError` | -| Permission Denied | No read permission | Raise `PermissionError` | -| File Too Large | Exceeds max_size limit | Raise `FileSizeError` | -| Encoding Error | Invalid file encoding | Raise `EncodingError` | -| Network Error | URL fetch failed | Raise `NetworkError` | - -## Security Considerations +The Text protocol handles various error conditions: -### Path Traversal Prevention - -```json -{ - "call_template_type": "text", - "file_path": "/safe/directory/${filename}", - "allowed_paths": ["/safe/directory/"] -} -``` +| Error Type | Condition | Behavior | +|------------|-----------|----------| +| File Not Found | File doesn't exist | Returns `RegisterManualResult` with `success: false` | +| Parse Error | Invalid JSON/YAML | Returns error result with exception details | +| Validation Error | Invalid UTCP manual | Returns error result with validation details | +| Generic Error | Unexpected exceptions | Returns error result with traceback | -### File Size Limits +### Error Response Format ```json { - "max_size": 1048576 // 1MB limit + "manual_call_template": { /* original template */ }, + "manual": { "tools": [] }, + "success": false, + "errors": ["Error details here..."] } ``` -### URL Restrictions +## Security Considerations -```json -{ - "allowed_domains": ["example.com", "docs.company.com"] -} -``` +- **Path Resolution**: Relative paths are resolved against the client's `root_dir` +- **Local Files Only**: Protocol only supports local file system access +- **No Authentication**: Text call templates don't support auth (always `None`) ## Best Practices -1. **Set Size Limits**: Always set appropriate max_size limits -2. **Validate Paths**: Validate file paths to prevent directory traversal -3. **Handle Encoding**: Specify encoding explicitly for non-UTF-8 files -4. **Use Line Ranges**: Use line ranges for large files to improve performance -5. **Pattern Efficiency**: Use efficient regex patterns for content filtering -6. **Cache Results**: Cache frequently accessed files -7. **Monitor Access**: Log file access for security auditing - -## Advanced Features - -### Conditional Reading - -```json -{ - "call_template_type": "text", - "file_path": "/var/log/app.log", - "condition": { - "modified_since": "2024-01-15T00:00:00Z" - } -} -``` - -### Multi-file Reading - -```json -{ - "call_template_type": "text", - "file_paths": [ - "/etc/app/config1.txt", - "/etc/app/config2.txt" - ], - "merge_strategy": "concatenate" -} -``` - -### Streaming Large Files - -```json -{ - "call_template_type": "text", - "file_path": "/var/log/huge.log", - "streaming": true, - "chunk_size": 8192 -} -``` +1. **Use Absolute Paths**: When possible, use absolute file paths for clarity +2. **Validate Files**: Ensure UTCP manual files are valid before registration +3. **Handle Errors**: Check `RegisterManualResult.success` before using tools +4. **Organize Manuals**: Keep manual files in a dedicated directory structure +5. **Version Control**: Include manual files in version control for consistency ## Common Use Cases -- **Configuration Management**: Reading config files, environment files -- **Log Analysis**: Processing application logs, system logs -- **Documentation**: Accessing README files, API docs, manuals -- **Data Processing**: Reading CSV, JSON, XML text files -- **Template Processing**: Reading template files for generation -- **Code Analysis**: Reading source code files for analysis -- **Monitoring**: Reading status files, health check files - -## Performance Considerations - -| File Size | Recommended Approach | -|-----------|---------------------| -| < 1MB | Read entire file | -| 1MB - 10MB | Use line ranges | -| 10MB - 100MB | Use streaming | -| > 100MB | Use external tools | +- **Static Tool Definitions**: Distributing tool configurations as files +- **Local Development**: Testing UTCP tools without external dependencies +- **Offline Environments**: Using tools in environments without network access +- **Configuration Management**: Reading tool definitions from config files +- **OpenAPI Integration**: Converting existing OpenAPI specs to UTCP tools -## Integration Examples +## API Reference -### With HTTP Protocol - -```json -{ - "name": "process_uploaded_file", - "description": "Process uploaded text file", - "inputs": { - "type": "object", - "properties": { - "file_url": {"type": "string"} - } - }, - "tool_call_template": { - "call_template_type": "text", - "file_path": "${file_url}", - "max_size": 5242880 - } -} -``` - -### With CLI Protocol - -```json -{ - "name": "analyze_log_file", - "description": "Analyze log file with external tool", - "inputs": { - "type": "object", - "properties": { - "log_path": {"type": "string"} - } - }, - "tool_call_template": { - "call_template_type": "cli", - "command": "log-analyzer", - "args": ["--file", "${log_path}", "--format", "json"] - } -} -``` +For detailed information about the implementation, see: +- [Text Call Template API](../api/plugins/communication_protocols/text/src/utcp_text/text_call_template.md) +- [Text Communication Protocol API](../api/plugins/communication_protocols/text/src/utcp_text/text_communication_protocol.md) diff --git a/versioned_docs/version-1.0/security.md b/versioned_docs/version-1.0/security.md index fbf10f6..14fa21f 100644 --- a/versioned_docs/version-1.0/security.md +++ b/versioned_docs/version-1.0/security.md @@ -118,29 +118,6 @@ Secure your UTCP manual endpoints: - ✅ Include request tracking headers - ✅ Implement retry limits -### WebSocket Security - -**Secure WebSocket configuration:** -```json -{ - "call_template_type": "websocket", - "url": "wss://api.example.com/ws", - "headers": { - "Authorization": "Bearer ${WS_TOKEN}", - "Origin": "https://trusted-domain.com" - }, - "ping_interval": 30, - "connection_timeout": 10 -} -``` - -**Security measures:** -- ✅ Use WSS (secure WebSocket) only -- ✅ Validate Origin headers -- ✅ Implement connection timeouts -- ✅ Use heartbeat/ping for connection health -- ✅ Limit concurrent connections per client - ### CLI Security :::danger High Risk Protocol @@ -427,7 +404,6 @@ Implement automated security validation: ### Protocol-Specific Security - [ ] **HTTP**: HTTPS only, certificate validation -- [ ] **WebSocket**: WSS only, origin validation - [ ] **CLI**: Sandboxed execution, input sanitization - [ ] **SSE**: Authenticated connections, event limits - [ ] **Text**: Path validation, size limits @@ -437,7 +413,6 @@ By following these security guidelines, you can safely implement UTCP while main For protocol-specific security details, see: - [HTTP Security](./protocols/http.md#security-considerations) -- [WebSocket Security](./protocols/websocket.md#security-considerations) - [CLI Security](./protocols/cli.md#security-considerations) - [SSE Security](./protocols/sse.md#security-considerations) - [Text Security](./protocols/text.md#security-considerations)