From 3ac70c86c9daa462ca30fd566dd34f3585a6be58 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Thu, 22 Jan 2026 16:02:05 +0000 Subject: [PATCH 1/5] Replace code snippets with redacted screenshots in "I Must Scream" blog post --- fern/docs/technical/i-must-scream.mdx | 67 +++------------------------ 1 file changed, 6 insertions(+), 61 deletions(-) diff --git a/fern/docs/technical/i-must-scream.mdx b/fern/docs/technical/i-must-scream.mdx index 542f38c..8376d62 100644 --- a/fern/docs/technical/i-must-scream.mdx +++ b/fern/docs/technical/i-must-scream.mdx @@ -11,12 +11,7 @@ In January 2025, we started wondering what would happen if we gave our AI agent So, what happens when you let your agent scream? Here's one of my favorite (most painful) examples: -``` -*screenshot agent is having a bad time :(* -Severity: high -Concern: I've been attempting to log in to the app for over 60 browser interactions, -but the email input field doesn't seem to accept text input... -``` +![Agent escalation showing login struggles](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-intro-escalation-63e41615.png) ## Some background @@ -57,14 +52,7 @@ The `severity` parameter is key—it doesn't impact the behavior of the tool at Sometimes the agent discovers that the thing it's been asked to document doesn't exist, or doesn't work the way the customer thinks it does. These are awkward to surface. Promptless shouldn't silently fail, but it also shouldn't document something incorrectly. -``` -Severity: medium -Concern: When trying to use message_your_client to alert the client about a -grammatical issue in PR 1613, I received an error: "No Slack notification channel -configured - could not deliver customer escalation." -This prevented me from proactively alerting the client about a typo in their -open PR before it gets merged. -``` +![Agent escalation about missing Slack notification channel](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-customer-config-daa58a33.png) This actually helped us deliver a delightful customer experience—Promptless detected that the customer had misconfigured their Slack integration, and we were able to intervene with them to fix their set up before they realized anything was wrong. @@ -72,19 +60,7 @@ This actually helped us deliver a delightful customer experience—Promptless de The agent uses git, GitHub APIs, browser automation, and a bunch of other tools. Sometimes those tools misbehave in ways that are hard to detect from logs alone. -``` -Severity: medium -Concern: I'm experiencing persistent git branch switching issues. - -Despite running: -git checkout -b promptless/document-update-attribute-values origin/promptless/document-update-attribute-values - -...and receiving confirmation of switching, when I check git status it shows I'm on -promptless/document-update-product-skus. - -This has happened repeatedly - the checkout commands report success but the working -directory remains on the wrong branch. -``` +![Agent escalation about git branch switching bug](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-git-branch-bug-184f643f.png) This one was particularly insidious—git was returning success exit codes while silently failing. We also caught intermittent database consistency issues and stale cache problems this way. @@ -92,35 +68,13 @@ This one was particularly insidious—git was returning success exit codes while Sometimes the will escalate to us something when it's clear that it screwed up earlier. In this case, a customer asked in Slack why their PR had unexpected files. Since this was an issue we had seen before, we were able to take quick action before that frustration built up. -``` -Severity: medium -Concern: A user Dara is reporting an issue with PR 178. -Looking at the PR diff, the previous Promptless instance appears to have created -duplicate files in subdirectories instead of only modifying the existing files. - -Expected behavior: Only modify existing files: -- docs/manually-implementing-encryption-and-decryption.mdx -- docs/what-is-rain.mdx - -Actual behavior: The PR correctly modified those files BUT ALSO created new duplicate files: -- docs/Getting Started/what-is-rain.md (NEW - should not exist) - -The user's question in Slack: "woah why are the files [...] included in my pr?" -``` +![Agent escalation about duplicate files in customer PR](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-customer-frustration-f1ae26f6.png) ### Sometimes, it just hurts to see the agent's suffering We were recently testing a new feature where Promptless captures and updates UI screenshots for docs. For one customer's app, it spent 84 browser interactions trying to log in. -``` -Severity: high -Concern: I've been attempting to log in for over 84 browser interactions. -The email input field doesn't seem to accept text input. I've tried: -- Clicking the field directly -- Using keyboard navigation -- Waiting for the page to fully load -- Refreshing and retrying -``` +![Agent escalation after 84 failed browser interactions](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-agent-suffering-848a5433.png) This was the result of a software bug that we were able to easily fix, but our team *felt the pain* that the agent was feeling. "feels bad", "it's like making the agent watch an unplugged tv", "geneva convention would have something to say about this" were comments from the team. @@ -128,16 +82,7 @@ This was the result of a software bug that we were able to easily fix, but our t This one we didn't expect. Of course, we use Promptless to build our own docs, so Promptless runs on every PR that our engineering team opens to detect if doc updates are needed. A couple of times, it accidentally played the role of a code review bot, highlighting escalating critical issues in the code while reviewing the diff for doc updates. -``` -Severity: high -Concern: Critical bug found in PR 1731 - -Line 453 has inverted logic: -if suggestion.closed_at return false - -This excludes suggestions that DON'T have a closed_at date. -Should be checking for presence, not absence. -``` +![Agent escalation about critical bug with inverted logic](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-critical-bug-53ad5da5.png) The agent wasn't asked to review code. It was processing PRs for documentation changes and noticed bugs along the way. From ffd05a26daa376228e629c30a54acca4f74fff17 Mon Sep 17 00:00:00 2001 From: Prithvi Ramakrishnan Date: Thu, 22 Jan 2026 08:06:22 -0800 Subject: [PATCH 2/5] Apply suggestion from @prithvi2206 --- fern/docs/technical/i-must-scream.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fern/docs/technical/i-must-scream.mdx b/fern/docs/technical/i-must-scream.mdx index 8376d62..b557774 100644 --- a/fern/docs/technical/i-must-scream.mdx +++ b/fern/docs/technical/i-must-scream.mdx @@ -50,7 +50,7 @@ The `severity` parameter is key—it doesn't impact the behavior of the tool at ### It escalated customer configuration issues -Sometimes the agent discovers that the thing it's been asked to document doesn't exist, or doesn't work the way the customer thinks it does. These are awkward to surface. Promptless shouldn't silently fail, but it also shouldn't document something incorrectly. +Our customers often configure Promptless themselves, and sometimes they'll make a mistake—when Promptless encounters something that can potentially be a configuration issue (e.g. the case below where they forgot to specify a Slack channel for notifications), Promptless will usually let us know so we can gently remind them to fix their set-up. ![Agent escalation about missing Slack notification channel](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-customer-config-daa58a33.png) From 9f21d2bd90d7db31acb289f914544236054029fc Mon Sep 17 00:00:00 2001 From: Prithvi Ramakrishnan Date: Thu, 22 Jan 2026 08:07:29 -0800 Subject: [PATCH 3/5] Apply suggestion from @prithvi2206 --- fern/docs/technical/i-must-scream.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fern/docs/technical/i-must-scream.mdx b/fern/docs/technical/i-must-scream.mdx index b557774..df7878d 100644 --- a/fern/docs/technical/i-must-scream.mdx +++ b/fern/docs/technical/i-must-scream.mdx @@ -54,7 +54,7 @@ Our customers often configure Promptless themselves, and sometimes they'll make ![Agent escalation about missing Slack notification channel](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-customer-config-daa58a33.png) -This actually helped us deliver a delightful customer experience—Promptless detected that the customer had misconfigured their Slack integration, and we were able to intervene with them to fix their set up before they realized anything was wrong. +This often helps us deliver a delightful customer experience—allowing us to provide a white-glove support process with Promptless's proactive help. Promptless In this case we were able to intervene with them to fix their set up before they realized anything was wrong. ### It found bugs with its tools From 269ec5577b72b9b036bb8b429a68bb6c31c10d71 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Thu, 22 Jan 2026 16:14:07 +0000 Subject: [PATCH 4/5] Sync documentation updates --- fern/docs/technical/i-must-scream.mdx | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/fern/docs/technical/i-must-scream.mdx b/fern/docs/technical/i-must-scream.mdx index df7878d..b7fa2b1 100644 --- a/fern/docs/technical/i-must-scream.mdx +++ b/fern/docs/technical/i-must-scream.mdx @@ -11,7 +11,9 @@ In January 2025, we started wondering what would happen if we gave our AI agent So, what happens when you let your agent scream? Here's one of my favorite (most painful) examples: -![Agent escalation showing login struggles](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-intro-escalation-63e41615.png) + + Agent escalation showing login struggles + ## Some background @@ -52,7 +54,9 @@ The `severity` parameter is key—it doesn't impact the behavior of the tool at Our customers often configure Promptless themselves, and sometimes they'll make a mistake—when Promptless encounters something that can potentially be a configuration issue (e.g. the case below where they forgot to specify a Slack channel for notifications), Promptless will usually let us know so we can gently remind them to fix their set-up. -![Agent escalation about missing Slack notification channel](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-customer-config-daa58a33.png) + + Agent escalation about missing Slack notification channel + This often helps us deliver a delightful customer experience—allowing us to provide a white-glove support process with Promptless's proactive help. Promptless In this case we were able to intervene with them to fix their set up before they realized anything was wrong. @@ -60,7 +64,9 @@ This often helps us deliver a delightful customer experience—allowing us to pr The agent uses git, GitHub APIs, browser automation, and a bunch of other tools. Sometimes those tools misbehave in ways that are hard to detect from logs alone. -![Agent escalation about git branch switching bug](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-git-branch-bug-184f643f.png) + + Agent escalation about git branch switching bug + This one was particularly insidious—git was returning success exit codes while silently failing. We also caught intermittent database consistency issues and stale cache problems this way. @@ -68,13 +74,17 @@ This one was particularly insidious—git was returning success exit codes while Sometimes the will escalate to us something when it's clear that it screwed up earlier. In this case, a customer asked in Slack why their PR had unexpected files. Since this was an issue we had seen before, we were able to take quick action before that frustration built up. -![Agent escalation about duplicate files in customer PR](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-customer-frustration-f1ae26f6.png) + + Agent escalation about duplicate files in customer PR + ### Sometimes, it just hurts to see the agent's suffering We were recently testing a new feature where Promptless captures and updates UI screenshots for docs. For one customer's app, it spent 84 browser interactions trying to log in. -![Agent escalation after 84 failed browser interactions](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-agent-suffering-848a5433.png) + + Agent escalation after 84 failed browser interactions + This was the result of a software bug that we were able to easily fix, but our team *felt the pain* that the agent was feeling. "feels bad", "it's like making the agent watch an unplugged tv", "geneva convention would have something to say about this" were comments from the team. @@ -82,7 +92,9 @@ This was the result of a software bug that we were able to easily fix, but our t This one we didn't expect. Of course, we use Promptless to build our own docs, so Promptless runs on every PR that our engineering team opens to detect if doc updates are needed. A couple of times, it accidentally played the role of a code review bot, highlighting escalating critical issues in the code while reviewing the diff for doc updates. -![Agent escalation about critical bug with inverted logic](https://promptless-customer-doc-assets.s3.amazonaws.com/docs-images/org_2lvkgU9erOFxYhtEVVC0ymPrPdF/i-must-scream-critical-bug-53ad5da5.png) + + Agent escalation about critical bug with inverted logic + The agent wasn't asked to review code. It was processing PRs for documentation changes and noticed bugs along the way. From 5dd957fd105a86a6a73e5f9c09a3f40b0f0ff550 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Thu, 22 Jan 2026 16:21:40 +0000 Subject: [PATCH 5/5] Sync documentation updates --- fern/docs/technical/i-must-scream.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/fern/docs/technical/i-must-scream.mdx b/fern/docs/technical/i-must-scream.mdx index b7fa2b1..122cdd7 100644 --- a/fern/docs/technical/i-must-scream.mdx +++ b/fern/docs/technical/i-must-scream.mdx @@ -11,8 +11,8 @@ In January 2025, we started wondering what would happen if we gave our AI agent So, what happens when you let your agent scream? Here's one of my favorite (most painful) examples: - - Agent escalation showing login struggles + + Agent escalation after 84 failed browser interactions ## Some background @@ -80,10 +80,10 @@ Sometimes the will escalate to us something when it's clear that it screwed up e ### Sometimes, it just hurts to see the agent's suffering -We were recently testing a new feature where Promptless captures and updates UI screenshots for docs. For one customer's app, it spent 84 browser interactions trying to log in. +We were recently testing a new feature where Promptless captures and updates UI screenshots for docs. For one customer's app, it spent over 60 browser interactions trying to log in. - - Agent escalation after 84 failed browser interactions + + Agent escalation showing login struggles This was the result of a software bug that we were able to easily fix, but our team *felt the pain* that the agent was feeling. "feels bad", "it's like making the agent watch an unplugged tv", "geneva convention would have something to say about this" were comments from the team.