diff --git a/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md b/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md index 87971bec4d5..5c9f1581e6f 100644 --- a/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md +++ b/app/_how-tos/ai-gateway/get-started-with-ai-gateway.md @@ -2,47 +2,146 @@ title: Get started with {{site.ai_gateway}} content_type: how_to permalink: /ai-gateway/get-started/ -description: Learn how to quickly get started with {{site.ai_gateway}} +description: Learn how to proxy LLM traffic with {{site.ai_gateway}} entities in {{site.konnect_product_name}} products: - - ai-gateway + - ai-gateway works_on: - - konnect + - konnect + +entities: + - ai-provider + - ai-model tags: - - get-started - - ai - - openai + - get-started + - ai tldr: - q: What is {{site.ai_gateway}}, and how can I get started with it? + q: How do I proxy LLM traffic with {{site.ai_gateway}} entities? a: | - With {{site.ai_gateway}}, you can deploy AI infrastructure for traffic - that is sent to one or more LLMs. + {{site.ai_gateway}} provides first-class entities for managing LLM providers and models in {{site.konnect_product_name}}. + Create an [AI Provider](/ai-gateway/entities/ai-provider/) entity to connect and authenticate to an LLM service like OpenAI, then create an [AI + Model](/ai-gateway/entities/ai-model/) entity to specify which model is available for requests. + + This tutorial shows you how to set up an AI Provider and AI Model for OpenAI in {{site.konnect_product_name}} using the {{site.konnect_short_name}} API and how to proxy your first request to OpenAI. tools: - - deck + - konnect-api prereqs: inline: - - title: OpenAI + - title: OpenAI credentials content: | - This tutorial uses the AI Proxy plugin with OpenAI. You'll need to [create an OpenAI account](https://auth.openai.com/create-account) and [get an API key](https://platform.openai.com/api-keys). Once you have your API key, create an environment variable: + This tutorial uses OpenAI as the LLM provider. You'll need to [create an OpenAI account](https://auth.openai.com/create-account) + and [get an API key](https://platform.openai.com/api-keys). Save your API key for the next steps: ```sh export OPENAI_API_KEY='' ``` - cleanup: inline: - - title: Destroy the {{site.ai_gateway}} container + - title: Clean up {{site.ai_gateway}} resources include_content: cleanup/products/ai-gateway - icon_url: /assets/icons/ai-gateway.svg min_version: - ai-gateway: '2.0' + ai-gateway: '2.0' + --- -## Placeholder +## Create an AI Provider entity + +Create an [AI Provider](/ai-gateway/entities/ai-provider/) entity to define your connection to OpenAI and store your authentication credentials: + + +{% konnect_api_request %} +url: /v1/ai-gateways/$AI_GATEWAY_ID/providers +status_code: 201 +method: POST +headers: + - 'Content-Type: application/json' + - 'Accept: application/json, application/problem+json' +body: + type: openai + display_name: generic-openai + name: generic-openai + config: + auth: + type: basic + headers: + - name: Authorization + value: Bearer $OPENAI_API_KEY +{% endkonnect_api_request %} + + +In this example, we're setting up the AI Provider with: + +* `type: openai`: Specifies that this provider connects to the OpenAI service using OpenAI's standard API format. +* `name: generic-openai`: A unique identifier that AI Models will reference to route requests through this provider. +* `config.auth`: Stores your OpenAI API key. {{site.ai_gateway}} securely manages this credential and injects it into upstream requests automatically, eliminating the need for clients to pass API keys. + +## Create an AI Model entity + +Create an [AI Model](/ai-gateway/entities/ai-model/) entity to declare which upstream models are available, configure how client requests are routed, and specify which AI Provider to use: + + +{% konnect_api_request %} +url: /v1/ai-gateways/$AI_GATEWAY_ID/models +status_code: 201 +method: POST +headers: + - 'Content-Type: application/json' + - 'Accept: application/json, application/problem+json' +body: + display_name: my-gpt-4o + name: my-gpt-4o + type: model + formats: + - type: openai + config: + route: + paths: + - /v1 + model: {} + logging: + payloads: false + statistics: true + targets: + - name: gpt-4o + provider: generic-openai + config: + type: openai + policies: [] + capabilities: + - generate +{% endkonnect_api_request %} + + +In this example, we're setting up the AI Model with: + +* `type: model`: Specifies this is a synchronous model for request/response workloads. +* `name: my-gpt-4o`: A unique identifier for this model. +* `formats: [type: openai]`: Declares that this model accepts requests in OpenAI-compatible format. +* `config.route.paths: [/v1]`: Configures the custom base path where this model's Routes will be accessible. Clients will send requests to paths that combine this base path with capability-specific Routes. +* `capabilities: [generate]`: Enables the text generation capability. The `generate` capability creates a `/chat/completions` endpoint, so combined with your base path, clients send chat requests to `/v1/chat/completions`. +* `targets`: Specifies which upstream AI Provider model to route requests to. Here, `provider: generic-openai` references the AI Provider we created earlier, and `name: gpt-4o` specifies which OpenAI model to call upstream. +* `config.logging`: Configures what gets logged. With `statistics: true`, usage metrics (tokens, latency, cost) are logged for monitoring and billing. With `payloads: false`, full request/response bodies are not logged for privacy. + +## Validate + +Send a chat request to verify your setup: -lorem ipsum \ No newline at end of file + +{% validation request-check %} +url: /v1/chat/completions +status_code: 200 +method: POST +headers: + - 'Accept: application/json' + - 'Content-Type: application/json' +body: + messages: + - role: "user" + content: "Say this is a test!" +{% endvalidation %} + diff --git a/app/_includes/cleanup/products/ai-gateway.md b/app/_includes/cleanup/products/ai-gateway.md index db895d7a01f..77a38069239 100644 --- a/app/_includes/cleanup/products/ai-gateway.md +++ b/app/_includes/cleanup/products/ai-gateway.md @@ -1,3 +1,5 @@ +To clean up all {{site.ai_gateway}} resources created in this guide, run: + ```bash curl -Ls https://get.konghq.com/quickstart/ai | bash -s -- -d ``` \ No newline at end of file diff --git a/app/_includes/prereqs/products/ai-gateway.md b/app/_includes/prereqs/products/ai-gateway.md index cd7bcced533..51a7b72643f 100644 --- a/app/_includes/prereqs/products/ai-gateway.md +++ b/app/_includes/prereqs/products/ai-gateway.md @@ -1,8 +1,34 @@ -{% assign summary='{{site.ai_gateway_name}} running' %} +{% assign summary='{{site.ai_gateway}} running' %} {% capture details_content %} -Placeholder prereq + +This is a {{site.konnect_short_name}} tutorial and requires a {{site.konnect_short_name}} personal access token. + +1. Create a new personal access token by opening the [{{site.konnect_short_name}} PAT page](https://cloud.konghq.com/global/account/tokens) and selecting **Generate Token**. + +1. Export your token to an environment variable: + + ```bash + export KONNECT_TOKEN='YOUR_KONNECT_PAT' + ``` + +1. Run the {{site.ai_gateway}} [quickstart script](https://get.konghq.com/quickstart/ai) to automatically provision a control plane and data plane in {{site.konnect_product_name}}, and configure your environment: + + ```bash + curl -Ls https://get.konghq.com/quickstart/ai | bash -s -- -k $KONNECT_TOKEN + ``` + +This sets up a {{site.ai_gateway}} control plane named `ai-quickstart`, provisions a local data plane, and prints out the following environment variables export: + ```bash -curl -Ls https://get.konghq.com/quickstart/ai | bash -s -- -d +export AI_GATEWAY_ID=your-gateway-id +export DECK_KONNECT_TOKEN=$KONNECT_TOKEN +export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart +export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com +export KONNECT_PROXY_URL='http://localhost:8000' ``` + +Copy and paste these into your terminal to configure your session. + {% endcapture %} + {% include how-tos/prereq_cleanup_item.html summary=summary details_content=details_content icon_url='/assets/icons/ai-gateway.svg' %}