From 302c9034780ca302de90099524bab41d2952fabf Mon Sep 17 00:00:00 2001 From: "inkeep[bot]" <257615677+inkeep[bot]@users.noreply.github.com> Date: Thu, 2 Jul 2026 17:45:08 +0000 Subject: [PATCH 1/2] Add AI observability signal source to inbox sources docs Reflects PR #67912 which adds a new evaluation_report signal source type for AI observability. The source is opt-in and lets teams receive inbox signals when evaluation report runs complete. --- contents/docs/self-driving/inbox/sources.mdx | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/contents/docs/self-driving/inbox/sources.mdx b/contents/docs/self-driving/inbox/sources.mdx index 9593687d1d32..b183eb5efbeb 100644 --- a/contents/docs/self-driving/inbox/sources.mdx +++ b/contents/docs/self-driving/inbox/sources.mdx @@ -5,8 +5,8 @@ sidebar: Docs showTitle: true --- -import { CallToAction } from 'components/CallToAction' -import List from 'components/List' +import { CallToAction } from "components/CallToAction"; +import List from "components/List"; @@ -14,7 +14,7 @@ Self-driving is currently in open beta. -Signal sources are what fill your inbox. A [signal](/docs/self-driving/signals) can be a production error, a support conversation, a Session Replay pattern, a log alert changing state, a Replay Vision scanner finding, or an issue from an external tracker. Each source watches one of these streams and turns what it finds into work worth investigating. +Signal sources are what fill your inbox. A [signal](/docs/self-driving/signals) can be a production error, a support conversation, a Session Replay pattern, a log alert changing state, a Replay Vision scanner finding, an AI evaluation report, or an issue from an external tracker. Each source watches one of these streams and turns what it finds into work worth investigating. ## Configure sources @@ -75,6 +75,13 @@ These sources are available when the matching PostHog product has data in your p description: "Outdated SDKs, proxy gaps, ingestion warnings, broken data models, and more.", lineClamp: 2, }, + { + label: "AI observability", + url: "/docs/ai-evals/evaluations", + icon: "IconAI", + description: "Findings from evaluation reports on your LLM traffic.", + lineClamp: 2, + }, ]} /> From 1528e632087a35e4c371e27cc66870406fab77ac Mon Sep 17 00:00:00 2001 From: "inkeep[bot]" <257615677+inkeep[bot]@users.noreply.github.com> Date: Thu, 2 Jul 2026 17:46:05 +0000 Subject: [PATCH 2/2] docs(ai-evals): add inbox signals subsection for opt-in eval report signals --- contents/docs/ai-evals/index.mdx | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/contents/docs/ai-evals/index.mdx b/contents/docs/ai-evals/index.mdx index 3a8c08077a61..6a4e016983da 100644 --- a/contents/docs/ai-evals/index.mdx +++ b/contents/docs/ai-evals/index.mdx @@ -17,13 +17,13 @@ Evaluations automatically assess the quality of your LLM [generations](/docs/ai- ## Choosing an evaluation type -| | LLM-as-a-judge | Code-based (Hog) | Sentiment analysis | -| --------------- | ------------------------------------------------------------ | ---------------------------------------------------------- | -------------------------------------------------------- | -| **Best for** | Subjective quality checks (tone, helpfulness, hallucination) | Deterministic rule-based checks (format, keywords, length) | Tracking how users feel during conversations | -| **Cost** | LLM API call per evaluation | Free | Free | -| **Speed** | Seconds | Milliseconds | Milliseconds | -| **Output** | Pass/fail with reasoning | Pass/fail with reasoning | Positive, neutral, or negative with a confidence score | -| **Setup** | Write a prompt | Write Hog code | Pick the type — no prompt or code required | +| | LLM-as-a-judge | Code-based (Hog) | Sentiment analysis | +| ------------ | ------------------------------------------------------------ | ---------------------------------------------------------- | ------------------------------------------------------ | +| **Best for** | Subjective quality checks (tone, helpfulness, hallucination) | Deterministic rule-based checks (format, keywords, length) | Tracking how users feel during conversations | +| **Cost** | LLM API call per evaluation | Free | Free | +| **Speed** | Seconds | Milliseconds | Milliseconds | +| **Output** | Pass/fail with reasoning | Pass/fail with reasoning | Positive, neutral, or negative with a confidence score | +| **Setup** | Write a prompt | Write Hog code | Pick the type — no prompt or code required | ## LLM-as-a-judge evaluations @@ -351,6 +351,12 @@ You can adjust the report configuration after creating an evaluation: - Add email or Slack delivery targets to receive notifications when reports are generated - Disable the report if you don't need automated summaries +### Inbox signals + +Evaluation reports can appear as signals in your [inbox](/docs/self-driving/inbox). To receive these signals, enable the **AI observability** source in your inbox signal sources settings. + +When enabled, the inbox receives a signal each time a report is generated, letting you review evaluation trends alongside other product signals. This is opt-in and disabled by default. + ## Evaluation status and errors Each evaluation has a status that indicates whether it's running: @@ -363,10 +369,10 @@ Each evaluation has a status that indicates whether it's running: PostHog automatically disables evaluations and sets them to error status when: -| Error reason | Cause | How to resolve | -| ------------------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| Error reason | Cause | How to resolve | +| ------------------------ | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Trial limit reached** | The 100 free trial evaluation runs have been exhausted. | Add your own API key in [**Settings** > **AI Observability**](https://app.posthog.com/settings/environment-ai-observability#ai-observability-byok). | -| **Model not allowed** | The selected model isn't available on the trial plan. | Switch to a trial-allowed model, or add your own provider API key. | +| **Model not allowed** | The selected model isn't available on the trial plan. | Switch to a trial-allowed model, or add your own provider API key. | | **Provider key deleted** | The provider API key used by this evaluation was removed. | Add a new provider API key in [**Settings** > **AI Observability**](https://app.posthog.com/settings/environment-ai-observability#ai-observability-byok). | ### Identifying errored evaluations