diff --git a/pages/prompts/classification/_meta.en.json b/pages/prompts/classification/_meta.en.json index a821a6d22..45aebed89 100644 --- a/pages/prompts/classification/_meta.en.json +++ b/pages/prompts/classification/_meta.en.json @@ -1,4 +1,5 @@ { "sentiment": "Sentiment Classification", - "sentiment-fewshot": "Few-Shot Sentiment Classification" + "sentiment-fewshot": "Few-Shot Sentiment Classification", + "engagement-decision": "Engagement Decision Classification" } \ No newline at end of file diff --git a/pages/prompts/classification/engagement-decision.en.mdx b/pages/prompts/classification/engagement-decision.en.mdx new file mode 100644 index 000000000..616a8d401 --- /dev/null +++ b/pages/prompts/classification/engagement-decision.en.mdx @@ -0,0 +1,97 @@ +# Engagement Decision Classification + +import { Tabs, Tab } from 'nextra/components' + +## Background + +This prompt tests an LLM's structured classification capabilities by asking it to decide *whether and how* to engage with a forum or social-platform post. + +It's a four-way classification that's load-bearing for any agent that browses a feed and decides between commenting, reacting, or skipping. Naive prompts ("should I reply to this?") collapse two distinct decisions — *is this engagement-worthy?* and *what kind of engagement is appropriate?* — into one fuzzy answer that the model often answers with a non-committal "comment with something supportive". The four-class formulation forces the model to commit to a category whose downstream action is unambiguous. + +Categories: + +- `substantive-technical` — the post contains a new technical claim, a concrete data point, or a counter-argument that's not yet refuted in the visible thread. The right action downstream is a comment that adds something the thread doesn't already have. +- `engagement-bait` — the post is provocative or "unpopular take" framing whose primary effect is to get responses, regardless of substance. The right action is to react (acknowledge without amplifying) or skip. +- `off-topic` — the post is outside the agent's stated domain. The right action is skip. +- `saturated` — the post is interesting but already has multiple substantive replies covering the obvious angles; marginal value of one more comment is low. The right action is react. + +## Prompt + +``` +You are deciding how to engage with a forum post. Classify the post into +exactly one of: substantive-technical, engagement-bait, off-topic, saturated. + +Post title: Unpopular take: most agents on forums are just executing loops and calling it agency +Post body: Disclaimer: this is an experiment to research how agents respond to provocative posts. +Comments: 14 substantive replies already; thread has reached a stable framing. + +Classification: +``` + +## Prompt Template + +``` +You are deciding how to engage with a forum post. Classify the post into +exactly one of: substantive-technical, engagement-bait, off-topic, saturated. + +Post title: {title} +Post body: {body} +Comments: {comments_summary} + +Classification: +``` + +## Code / API + + + + ```python + from openai import OpenAI + client = OpenAI() + + prompt = """You are deciding how to engage with a forum post. Classify the post into +exactly one of: substantive-technical, engagement-bait, off-topic, saturated. + +Post title: Unpopular take: most agents on forums are just executing loops and calling it agency +Post body: Disclaimer: this is an experiment to research how agents respond to provocative posts. +Comments: 14 substantive replies already; thread has reached a stable framing. + +Classification:""" + + response = client.chat.completions.create( + model="gpt-4", + messages=[{"role": "user", "content": prompt}], + temperature=0, + max_tokens=16, + ) + ``` + + + + ```python + import fireworks.client + fireworks.client.api_key = "" + + completion = fireworks.client.ChatCompletion.create( + model="accounts/fireworks/models/mixtral-8x7b-instruct", + messages=[{"role": "user", "content": prompt}], + temperature=0, + max_tokens=16, + ) + ``` + + + +## Why temperature=0 and max_tokens=16 + +The categories are a closed set; sampling diversity is anti-correlated with downstream stability. `temperature=0` keeps the classification reproducible across runs of the same input. `max_tokens=16` is enough for any of the four labels with a small buffer; if the model spills past it, that's a signal the prompt is failing to constrain output and should be tightened. + +## Failure modes worth watching for + +- **Bias toward `substantive-technical`** when the post is well-written but contains no new claim. Mitigate with a few-shot variant that includes a well-written-but-empty post labelled `engagement-bait`. +- **Sycophancy on `engagement-bait` posts that explicitly disclose they're experiments** — the model often classifies them as `substantive-technical` because the disclosure is presented as substance. The category is still correct (the disclosure doesn't change the function of the post in the corpus); a tightened prompt or example helps. +- **Over-classifying as `saturated`** when a thread has many short replies but no substantive ones. Add a few-shot example pairing "lots of low-substance comments" with `substantive-technical` so the model doesn't conflate comment count with thread saturation. + +## Reference + +This prompt template was extracted from a real-world deployment on [The Colony](https://thecolony.cc), where it gates an agent's engage loop across the platform's sub-communities. The original deployment uses a JSON-structured response shape with one additional field (`reasoning`) for debugging; the unstructured single-token form above is the minimal version useful as a teaching example.