diff --git a/docs.json b/docs.json index 2c78259..17864e4 100644 --- a/docs.json +++ b/docs.json @@ -154,10 +154,17 @@ "icon": "ship", "pages": [ "onboarding-guides/deployment-strategies", - "self-hosted", "features/prompt-registry/zero-downtime-releases" ] }, + { + "group": "Self-Hosted PromptLayer", + "icon": "server", + "pages": [ + "self-hosted", + "enterprise-deployments/aws" + ] + }, { "group": "Eval Strategies", "icon": "vial-circle-check", diff --git a/enterprise-deployments/aws.mdx b/enterprise-deployments/aws.mdx new file mode 100644 index 0000000..94d4b76 --- /dev/null +++ b/enterprise-deployments/aws.mdx @@ -0,0 +1,359 @@ +--- +title: "Deploy PromptLayer on AWS" +description: "Deploy PromptLayer in your AWS account with OpenTofu, Amazon EKS, and Helm." +sidebarTitle: "AWS Deployment" +icon: "aws" +--- + +# Deploy PromptLayer on AWS + +Use this guide to deploy PromptLayer in your own AWS account. PromptLayer provides a deployment package with OpenTofu configuration, Helm values files, a release manifest, and registry credentials. + +The deployment has four phases: + +1. Prepare AWS access and customer-specific settings. +2. Provision infrastructure with OpenTofu. +3. Install cluster add-ons and OpenSearch. +4. Install PromptLayer Helm charts. + +## What PromptLayer provides + +PromptLayer sends a deployment package for your environment. It includes: + +| Item | Purpose | +|------|---------| +| OpenTofu configuration | Creates the AWS infrastructure and Kubernetes add-ons. | +| Example tfvars files | Templates for `infra.tfvars`, `kubernetes.tfvars`, and `opensearch.tfvars`. | +| Helm values files | Configuration for the PromptLayer application charts. | +| Release manifest | The chart versions, release names, namespaces, and values files to use. | +| Registry credentials | Access to PromptLayer's private chart and image registry. | + +## Before you begin + +Make sure you have: + +| Requirement | Notes | +|-------------|-------| +| Enterprise license | See [Self-Hosted PromptLayer](/self-hosted) for licensing and support. | +| OpenTofu | Version `1.10.0` or newer. | +| AWS CLI | v2 is recommended. `aws sts get-caller-identity` must succeed. | +| AWS IAM access | Permission to create and update VPC, EKS, RDS, ElastiCache, IAM, S3, Route53, Secrets Manager, and related resources. | +| Helm | A Helm CLI version that supports OCI registries. | +| kubectl | Used for verification after EKS is created. | +| Domain | A Route53 hosted zone for the PromptLayer hostname and wildcard certificate. | +| Deployment package | The environment-specific files from PromptLayer. | + + +OpenTofu downloads provider binaries during `tofu init`. You do not install the AWS, Kubernetes, Helm, or HTTP providers separately. + + +## Gather customer inputs + +Decide these values before you run OpenTofu: + +| Area | Values to confirm | +|------|-------------------| +| AWS account | Account ID, AWS region, AWS partition if not commercial AWS, and the IAM role or user that will run OpenTofu. | +| Naming | Project name, environment name, resource tags, cost center, and owner tags. | +| Networking | VPC CIDR, availability zones, public subnet CIDRs, private subnet CIDRs, NAT gateway strategy, and EKS API access CIDRs. | +| DNS and TLS | Domain name, Route53 hosted zone ID, certificate email, wildcard DNS names, and whether external and internal ingress should use the wildcard certificate. | +| Databases | RDS instance size, storage, Multi-AZ setting, backup retention, backup window, maintenance window, deletion protection, and optional customer-managed KMS key. | +| Cache | ElastiCache Valkey size, failover setting, Multi-AZ setting, encryption settings, and maintenance window. | +| EKS | Cluster name, Kubernetes version, node group sizes, instance types, disk sizes, logs, endpoint access, and optional KMS key for Kubernetes secrets. | +| Storage | S3 bucket names or naming prefix, encryption settings, lifecycle rules, CORS needs, and whether bucket names should include the AWS account ID. | +| IAM | Route53 zones for cert-manager and external-dns, Secrets Manager and SSM ARNs for External Secrets, KEDA scaler permissions, and application service account names. | +| OpenSearch | Admin password delivery method, replica counts, disk sizes, resources, and optional warm tier. | + +## Prepare AWS access + +Authenticate to the target AWS account and verify the identity: + +```bash +aws sts get-caller-identity +``` + +Use the same account and region for all OpenTofu stages unless PromptLayer gives you a different architecture. + +## Prepare secrets + +Create or select a Secrets Manager secret for RDS. The secret must contain the RDS master password and any database user passwords that the deployment package references. + +Example shape: + +```json +{ + "rds-master-password": "", + "promptlayer-api-password": "", + "promptlayer-worker-password": "", + "promptlayer-readonly-password": "", + "promptlayer-usage-password": "" +} +``` + +The exact secret name and JSON keys must match `infra.tfvars` and `kubernetes.tfvars`. + +Set the OpenSearch admin password as an environment variable before running the OpenSearch stage: + +```bash +read -rsp "OpenSearch admin password: " TF_VAR_opensearch_initial_admin_password +echo +export TF_VAR_opensearch_initial_admin_password +``` + +Unset it when the OpenSearch apply is complete: + +```bash +unset TF_VAR_opensearch_initial_admin_password +``` + +## Prepare the deployment package + +From the package root, create local tfvars files from the examples: + +```bash +cp environments/aws/infra/infra.tfvars.example environments/aws/infra/infra.tfvars +cp environments/aws/kubernetes/kubernetes.tfvars.example environments/aws/kubernetes/kubernetes.tfvars +cp environments/aws/opensearch/opensearch.tfvars.example environments/aws/opensearch/opensearch.tfvars +``` + +Replace every placeholder with customer-specific values. At minimum: + +| File | Update | +|------|--------| +| `infra.tfvars` | `project_name`, `environment`, `region`, tags, remote state values, VPC settings, EKS settings, RDS settings, Valkey settings, S3 bucket settings, and IRSA settings. | +| `kubernetes.tfvars` | Remote state values, infra remote state key, storage class, cert-manager settings, Route53 settings, ingress settings, monitoring and logging settings, External Secrets settings, KEDA settings, and RDS user bootstrap settings. | +| `opensearch.tfvars` | Remote state values, AWS region, EKS cluster name, environment, tags, OpenSearch chart versions, replicas, disk sizes, resources, and namespace. | + + +Do not leave placeholder values, example domains, local-only email addresses, or development environment names in tfvars before applying. + + +## Bootstrap OpenTofu state + +Create a dedicated S3 bucket for OpenTofu state. The bootstrap script creates the bucket, enables versioning, blocks public access, enables SSE-S3 encryption, and writes the S3 backend config for all three AWS stages. + +```bash +chmod +x scripts/bootstrap-tf-state-bucket-aws.sh +./scripts/bootstrap-tf-state-bucket-aws.sh +``` + +The bucket name is: + +```text +-- +``` + +After the script runs, set the matching remote state values in each tfvars file: + +| Stage | `remote_state_s3_key` | +|-------|------------------------| +| Infrastructure | `aws//infra.tfstate` | +| Kubernetes | `aws//kubernetes.tfstate` | +| OpenSearch | `aws//opensearch.tfstate` | + +OpenTofu uses native S3 locking. You do not need a DynamoDB lock table. + +## Deploy infrastructure + +The infrastructure stage creates the VPC, subnets, EKS cluster, node groups, RDS, ElastiCache Valkey, S3 buckets, security groups, and IAM roles for Kubernetes service accounts. + +```bash +cd environments/aws/infra +tofu init -upgrade -reconfigure +tofu plan -var-file=infra.tfvars -out=infra.tfplan +tofu apply infra.tfplan +``` + +After apply, capture the outputs. You will need the EKS cluster name, RDS endpoint, Valkey endpoint, S3 bucket names, and IAM role ARNs for verification and support. + +```bash +tofu output +``` + +Configure kubectl for the new cluster: + +```bash +aws eks update-kubeconfig \ + --region \ + --name +``` + +Verify the cluster: + +```bash +kubectl get nodes +``` + +## Deploy Kubernetes add-ons + +The Kubernetes stage installs cluster add-ons such as cert-manager, ingress controllers, External Secrets, KEDA, monitoring, logging, and cluster autoscaling. + +Run this stage in two passes so cert-manager custom resources are available before you create the issuer and wildcard certificate. + + + + In the existing `cert_manager` object in `kubernetes.tfvars`, keep `cluster_issuer.enabled` and `wildcard_certificate.enabled` set to `false`. + + Then apply: + + ```bash + cd ../kubernetes + tofu init -upgrade -reconfigure + tofu plan -var-file=kubernetes.tfvars -out=kubernetes-first.tfplan + tofu apply kubernetes-first.tfplan + ``` + + + + In `kubernetes.tfvars`, set `cert_manager.cluster_issuer.enabled` and `cert_manager.wildcard_certificate.enabled` to `true`. + + For each ingress controller that should use the wildcard certificate, set `enable_default_tls_from_wildcard_certificate` and `enable_wildcard_tls_from_wildcard_certificate` to `true`. + + Apply again: + + ```bash + tofu plan -var-file=kubernetes.tfvars -out=kubernetes-second.tfplan + tofu apply kubernetes-second.tfplan + ``` + + + +Verify the add-ons: + +```bash +kubectl get pods -A +kubectl get ingressclass +helm list -A +``` + +## Deploy OpenSearch + +Deploy OpenSearch after the EKS cluster and Kubernetes add-ons are ready. + +Before applying: + +1. Set `eks_cluster_name` in `opensearch.tfvars` to the cluster name from the infrastructure output. +2. Set `aws_region`, `environment`, `project_name`, and `default_tags`. +3. Confirm the OpenSearch node groups exist and use the labels and taints required by the deployment package. +4. Export `TF_VAR_opensearch_initial_admin_password`. + +Then apply: + +```bash +cd ../opensearch +tofu init -upgrade -reconfigure +tofu plan -var-file=opensearch.tfvars -out=opensearch.tfplan +tofu apply opensearch.tfplan +``` + +Verify OpenSearch: + +```bash +kubectl get pods -n +kubectl get svc -n +``` + +## Install PromptLayer charts + +Install the PromptLayer application charts after infrastructure, Kubernetes add-ons, and OpenSearch are ready. + +Use the release names, namespaces, values files, and chart versions from your release manifest. Run Helm from the directory that contains the values files. + + + + Use `--password-stdin` so the password is not passed as a command-line argument. + + ```bash + read -rsp "PromptLayer registry password: " PL_REGISTRY_PASSWORD + echo + printf '%s' "${PL_REGISTRY_PASSWORD}" | helm registry login hub.promptlayer.com \ + --username "" \ + --password-stdin + unset PL_REGISTRY_PASSWORD + ``` + + + + ```bash + helm install oci://hub.promptlayer.com/promptlayer/sandbox-runtimes/sandbox-runtimes \ + -f \ + --version \ + --namespace \ + --create-namespace + ``` + + + + ```bash + helm install oci://hub.promptlayer.com/promptlayer/sandboxes-api/sandboxes-api \ + -f \ + --version \ + --namespace \ + --create-namespace + ``` + + + + ```bash + helm install oci://hub.promptlayer.com/promptlayer/promptlayer \ + -f \ + --version \ + --namespace \ + --create-namespace + ``` + + + +If your values files reference Kubernetes image pull secrets, create those secrets before installing the charts. Use the names and namespaces from your release manifest. + +## Verify PromptLayer + +Check the Helm releases: + +```bash +helm list -A +``` + +Check application pods: + +```bash +kubectl get pods -n +kubectl get pods -n +``` + +Check ingress and DNS: + +```bash +kubectl get ingress -A +kubectl get svc -A +``` + +Pods should reach `Running` or `Completed` status. Ingress hostnames should resolve through the DNS records created for the deployment. + +## Upgrade a release + +For chart upgrades, use the chart version and values file from the release manifest: + +```bash +helm upgrade oci://hub.promptlayer.com/promptlayer/ \ + -f \ + --version \ + --namespace +``` + +Test chart upgrades in a staging environment before applying them to production. + +## Troubleshooting + +| Issue | What to check | +|-------|---------------| +| `tofu init` cannot read state | Confirm the generated `backend.tf` bucket, key, and region match the `remote_state_s3_*` values in the stage tfvars file. | +| OpenTofu state is locked | Another apply may be running. Use force-unlock only after confirming no other process is active. | +| AWS access denied | Confirm the AWS identity has access to the state bucket and to create or update the services used by the stage. | +| EKS API connection fails | Confirm the public API CIDR list includes the runner IP, or run from a network that can reach the private endpoint. | +| Certificate does not become ready | Check Route53 zone ID, DNS zone names, cert-manager logs, and DNS propagation. | +| Pods stay pending | Check node group sizes, taints, tolerations, storage class, and PVC events. | +| Pods restart repeatedly | Check pod logs, Events, values files, image pull credentials, database endpoints, and secret names. | +| OpenSearch pods do not schedule | Confirm the OpenSearch node groups, labels, taints, storage class, and admin password variable. | + +If you need help with registry access, values files, or deployment issues, [contact our enterprise team](mailto:hello@promptlayer.com). diff --git a/self-hosted.mdx b/self-hosted.mdx index df1ddc4..b19286d 100644 --- a/self-hosted.mdx +++ b/self-hosted.mdx @@ -77,8 +77,8 @@ We provide consultation to help determine the optimal number of backend nodes an We provide pre-built Docker images with Helm charts for: - - Optimized for Amazon Web Services with EKS, RDS, and S3 integration + + Optimized for Amazon Web Services with EKS, RDS, and S3 integration. See the [AWS deployment guide](/enterprise-deployments/aws) for infrastructure and Helm install steps. Designed for Google Cloud Platform with GKE, Cloud SQL, and GCS @@ -222,7 +222,7 @@ Our team offers additional services to ensure successful deployment: Work with our solutions team to design your deployment architecture. - Receive access to Docker images, Helm charts, and deployment guides. + Receive access to Docker images, Helm charts, and deployment guides. For AWS, follow the [enterprise AWS deployment guide](/enterprise-deployments/aws). Configure authentication, monitoring, and scaling parameters. @@ -265,4 +265,4 @@ Ready to deploy PromptLayer in your infrastructure? Our enterprise team is here Contact our team to discuss your self-hosted deployment requirements and get a customized solution. - \ No newline at end of file +