diff --git a/src/content/changelog/ai-crawl-control/2025-12-11-pay-per-crawl-custom-terms.mdx b/src/content/changelog/ai-crawl-control/2025-12-11-pay-per-crawl-custom-terms.mdx new file mode 100644 index 00000000000000..9b8cca0459a04b --- /dev/null +++ b/src/content/changelog/ai-crawl-control/2025-12-11-pay-per-crawl-custom-terms.mdx @@ -0,0 +1,17 @@ +--- +title: Custom terms URL for Pay Per Crawl +description: Site owners can now provide a custom terms URL that will be included in HTTP responses to crawlers. +date: 2025-12-11 +--- + +:::note[Pay per crawl closed beta] +Pay per crawl is currently in closed beta. + +To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer. + +To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/). +::: + +Site owners using [Pay Per Crawl](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) can now optionally configure a **Terms URL** in **AI Crawl Control** > **Settings**. The URL is included in the `Link` header of HTTP 200 and 402 responses served to crawlers, allowing you to provide custom terms and conditions for crawler access to your paid content. + +Learn more about [setting a pay per crawl Terms URL](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/). diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx index ee80777b7db32d..8deaf59605848c 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx @@ -24,7 +24,6 @@ When an AI crawler makes a request to a page protected by pay per crawl, the ser ```txt HTTP/2 402 -date: Fri, 06 Jun 2025 08:42:38 GMT crawler-price: USD 0.01 ``` @@ -56,8 +55,7 @@ When you specify a header to indicate payment, you may receive one of two respon The value of the `crawler-charged` header indicates the exact amount that will be billed to your Cloudflare account for the request. ```txt -HTTP 200 -date: Fri, 06 Jun 2025 08:42:38 GMT +HTTP/2 200 crawler-charged: USD 0.01 ``` @@ -67,7 +65,6 @@ If the request is unsuccessful, you will receive an error response with a `crawl ```txt HTTP/2 402 -date: Fri, 06 Jun 2025 08:42:38 GMT content-type: text/plain; charset=utf-8 crawler-price: USD 0.01 crawler-error: InvalidCrawlerExactPrice @@ -75,6 +72,12 @@ crawler-error: InvalidCrawlerExactPrice Refer to [Pay Per Crawl error codes](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/error-codes/) for a complete list of error codes and troubleshooting guidance. +### 2.4. Check for custom terms + +Before using paid content, check if the response includes a `Link` header. If present, the site owner has provided custom terms and conditions that supersede the default Pay Per Crawl agreement. Review these terms before proceeding with payment. + +This header may be present in both HTTP 200 and 402 responses. + ## 3. Track your spending When you successfully access pay per crawl content (response with HTTP status code 200), the response will include `crawler-charged`. For example: diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx index a060311c3919bf..17634f74a9e6df 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx @@ -21,23 +21,34 @@ click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owne click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/" ``` -Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price and enable pay per crawl for that domain. +Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price, configure a Terms URL, and enable pay per crawl for that domain. {/* prettier-ignore */} 1. Go to **AI Crawl Control**. - + 2. Go to the **Settings** tab. 3. In the **Pay Per Crawl** card, select **Enable**. -4. Set your default per crawl price. This is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler. - - (Optional) To set different prices for different content, select **Enable custom pricing**. Refer to [Advanced configuration](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/advanced-configuration/#custom-pricing) for details. -5. Select **Save**. +4. Set your per crawl price - this is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler. +5. (Optional) Set a **Terms URL** - a valid URL to custom terms and conditions that will be included in the `Link` header of HTTP 200 and 402 responses to crawlers. +6. Select **Save**. -After enabling and setting a price, the domain's status in Account Settings will change to **Enabled**. +## Crawl price + +The crawl price is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler that is set to **Charge**. :::note[Pricing considerations] The minimum price is $0.01 USD per crawl. Consider your content value and expected crawler volume when setting your price. ::: + +After setting a crawl price the first time, the domain's status in Account Settings will update to **Enabled**. + +## Terms URL (optional) + +You can optionally provide a **Terms URL** that points to custom terms and conditions for crawlers paying to access to your content. When configured: + +- The URL is included in the `Link` header of HTTP 200 and 402 responses served to crawlers by Pay Per Crawl. +- Crawler operators are responsible for reviewing your custom terms before using your content.