Fix: add cold-start candidate guardrails

experiments/active/2026-07-14-cold-start-candidate-filter.md

@@ -0,0 +1,96 @@
+# Experiment: Cold Start Candidate Filter
+
+Created: 2026-07-14
+Status: active - implementation in PR
+Owner: ceo
+Implementer: CTO
+Implementation Issue: FFM-1882
+Verification Issue: FFM-1883
+Deployed: pending
+Measure after: 2026-07-28
+
+## Hypothesis
+
+True-new users are not primarily failing on the first meme. The July 13 first-10
+readout showed first-meme continuation at 87.5%, but only 58.9% of true-new
+users reached five sends and 26.8% reached ten. If the cold-start candidate
+pools exclude weak source slices for positions 2-10, reached-10 and first-10
+continuation should improve without changing the mature-user feed.
+
+## Changes Planned
+
+- Scope the treatment to true-new cold-start users only.
+- Add a feature-flagged treatment path for cold-start candidate selection.
+- Start with source-level guardrails from the July 13 readout:
+  `https://vk.com/dfzwe4`, `https://vk.com/eternalclassic`,
+  `https://t.me/ukrmemesmineproblemes`, and `https://t.me/hindi_jokes_desi_memes`.
+- Avoid a broad language reweight in v1. EN/RU mix should be measured as a
+  guardrail, not forced from a 56-user sample.
+- Preserve existing fallbacks so empty pools route to the current
+  `text_light_lr_smoothed` / `best_uploaded_memes` behavior rather than
+  starving users.
+- Keep `recently_liked_blender_v2` untouched because it is a mature-user
+  experiment with its own 2026-07-22 checkpoint.
+
+## Implementation Notes
+
+- Rollback: set `COLD_START_CANDIDATE_GUARDRAILS_ENABLED=false`.
+- Treatment attribution: guarded candidates are persisted as
+  `cold_start_explore_guarded` or `cold_start_adapt_guarded` in
+  `user_meme_reaction.recommended_by`.
+- Fallback rate: use recommendation diagnostics `fallback_used`, grouped by
+  `cold_start_candidate_guardrails_applied`.
+
+## Metrics Before
+
+2026-07-13 readout, 14-day true-new cohort:
+
+- Cohort users: 56.
+- First-meme LR: 41.4%; first-meme continuation: 87.5%.
+- First-10 LR: 39.5%; first-10 continuation: 69.8%; first-10 quality score: -0.179.
+- Reached 5 users: 33/56 (58.9%).
+- Reached 10 users: 15/56 (26.8%).
+- Second-session users: 31/56 (55.4%).
+- Position 2 continuation: 62.0%.
+- Positions 6-7 continuation: 51.9-52.2%.
+- 2026-07-14 health: North Star 21.0 vs previous 21.0, no severity incident;
+  `recently_liked_blender_v2` sample gate remains unmet at 357 control / 400 treatment.
+
+## Success Criteria
+
+- Treatment reached-10 improves from 26.8% to at least 35% after the
+  2026-07-28 checkpoint, or reaches a directionally positive lift with at least
+  30 treatment users.
+- Positions 2-5 continuation improves by at least 5pp versus baseline/control.
+- First-meme continuation does not fall below 84%.
+- First-10 LR does not fall by more than 3pp.
+- No product-health incident: North Star does not drop by more than 10% week
+  over week, stats remain fresh, and candidate pool fallback rate does not spike.
+
+## Failure Criteria
+
+- Treatment reached-10 is flat or worse after the checkpoint with enough users.
+- First-meme continuation drops below 84% or first-10 LR drops by more than 3pp.
+- Cold-start pool exhaustion causes fallback-heavy recommendations or visible
+  delivery issues.
+- The implementation changes mature-user routing, `recently_liked_blender_v2`,
+  moderator low-sent quota, or unrelated engines.
+
+## Guardrails
+
+- Feature flag or equivalent fast rollback path required before production exposure.
+- Report source/language mix, fallback rate, first-10 positions, reached-5,
+  reached-10, second-session users, first-meme continuation, first-10 LR, and
+  first-10 continuation.
+- Preferred Analyst checkpoint artifact:
+  `experiments/reports/2026-07-28-cold-start-candidate-filter-readout.md`.
+- No Comms post until the experiment produces a user-facing result, not just an
+  internal filter launch.
+
+## Metrics After
+
+Pending 2026-07-28 readout.
+
+## Conclusion
+
+Pending.

src/config.py

@@ -55,6 +55,9 @@ class Config(BaseSettings):
     # users (nsessions <= 1). Dormant returners with nmemes_sent < 30 but
     # multiple sessions fall through to the growing-user blender instead.
     COLD_START_NSESSIONS_GATE_ENABLED: bool = True
+    # FFM-1882: narrow true-new cold-start positions 2-10 experiment.
+    # Roll back by setting this to false in production env.
+    COLD_START_CANDIDATE_GUARDRAILS_ENABLED: bool = True
 
     # Recommendation batch diagnostics are realtime operational data, not
     # durable product facts. Keep compact logs/spans always on and sample full

src/recommendations/candidates.py

@@ -16,6 +16,16 @@
 GOAT_MIN_AGE_DAYS = 3  # days since created_at — must have aged enough to accumulate reactions
 GOAT_RECENTLY_SENT_WINDOW_DAYS = 30
 TEXT_LIGHT_MAX_OCR_WORDS = 30
+COLD_START_GUARDRAIL_SOURCE_URLS = (
+    "https://vk.com/dfzwe4",
+    "https://vk.com/eternalclassic",
+    "https://t.me/ukrmemesmineproblemes",
+    "https://t.me/hindi_jokes_desi_memes",
+)
+COLD_START_EXPLORE_RECOMMENDED_BY = "cold_start_explore"
+COLD_START_ADAPT_RECOMMENDED_BY = "cold_start_adapt"
+COLD_START_EXPLORE_GUARDED_RECOMMENDED_BY = "cold_start_explore_guarded"
+COLD_START_ADAPT_GUARDED_RECOMMENDED_BY = "cold_start_adapt_guarded"
 
 _OCR_TEXT_SQL = "trim(coalesce(M.ocr_result->>'text', M.ocr_result->'raw_result'->>'ocr_text', ''))"
 TEXT_LIGHT_OCR_FILTER_SQL = f"""
@@ -26,6 +36,9 @@
                 END
             ) <= :text_light_max_ocr_words
 """
+COLD_START_GUARDRAIL_SOURCE_FILTER_SQL = """
+            AND NOT (S.url = ANY(:cold_start_guardrail_source_urls))
+"""
 
 
 def _build_params(
@@ -41,6 +54,38 @@ def _build_params(
     return params
 
 
+def _cold_start_recommended_by(engine: str, candidate_guardrails_enabled: bool) -> str:
+    if engine == COLD_START_EXPLORE_RECOMMENDED_BY and candidate_guardrails_enabled:
+        return COLD_START_EXPLORE_GUARDED_RECOMMENDED_BY
+    if engine == COLD_START_ADAPT_RECOMMENDED_BY and candidate_guardrails_enabled:
+        return COLD_START_ADAPT_GUARDED_RECOMMENDED_BY
+    return engine
+
+
+def _cold_start_guardrail_source_filter(candidate_guardrails_enabled: bool) -> str:
+    return COLD_START_GUARDRAIL_SOURCE_FILTER_SQL if candidate_guardrails_enabled else ""
+
+
+def _cold_start_params(
+    user_id: int,
+    limit: int,
+    exclude_meme_ids: list[int],
+    *,
+    engine: str,
+    candidate_guardrails_enabled: bool,
+) -> dict[str, Any]:
+    params = _build_params(
+        user_id,
+        limit,
+        exclude_meme_ids,
+        recommended_by=_cold_start_recommended_by(engine, candidate_guardrails_enabled),
+        text_light_max_ocr_words=TEXT_LIGHT_MAX_OCR_WORDS,
+    )
+    if candidate_guardrails_enabled:
+        params["cold_start_guardrail_source_urls"] = list(COLD_START_GUARDRAIL_SOURCE_URLS)
+    return params
+
+
 async def best_uploaded_memes(
     user_id: int,
     limit: int = 10,
@@ -413,6 +458,7 @@ async def cold_start_explore(
     user_id: int,
     limit: int = 5,
     exclude_meme_ids: list[int] = [],
+    candidate_guardrails_enabled: bool = False,
 ):
     """Phase 1 cold start: quality-first selection for new user first impression.
 
@@ -436,12 +482,14 @@ async def cold_start_explore(
         SELECT
             M.id
             , M.type, M.telegram_file_id, M.caption
-            , 'cold_start_explore' AS recommended_by
+            , :recommended_by AS recommended_by
             , COALESCE(MS.nlikes, 0) AS nlikes
 
         FROM meme M
         INNER JOIN meme_stats MS
             ON MS.meme_id = M.id
+        INNER JOIN meme_source S
+            ON S.id = M.meme_source_id
         INNER JOIN user_language L
             ON L.language_code = M.language_code
             AND L.user_id = :user_id
@@ -455,6 +503,7 @@ async def cold_start_explore(
             AND (MS.nlikes + MS.ndislikes) >= 20
             AND MS.lr_smoothed >= 0.10
             {TEXT_LIGHT_OCR_FILTER_SQL}
+            {_cold_start_guardrail_source_filter(candidate_guardrails_enabled)}
             {exclude_meme_ids_sql_filter(exclude_meme_ids)}
 
         ORDER BY (MS.nlikes::float / NULLIF(MS.nlikes + MS.ndislikes, 0)) DESC,
@@ -463,11 +512,12 @@ async def cold_start_explore(
     """
     return await fetch_all(
         text(query),
-        _build_params(
+        _cold_start_params(
             user_id,
             limit,
             exclude_meme_ids,
-            text_light_max_ocr_words=TEXT_LIGHT_MAX_OCR_WORDS,
+            engine=COLD_START_EXPLORE_RECOMMENDED_BY,
+            candidate_guardrails_enabled=candidate_guardrails_enabled,
         ),
     )
 
@@ -476,6 +526,7 @@ async def cold_start_adapt(
     user_id: int,
     limit: int = 15,
     exclude_meme_ids: list[int] = [],
+    candidate_guardrails_enabled: bool = False,
 ):
     """Phase 2 cold start: adapt to user's reactions in real-time.
 
@@ -506,12 +557,14 @@ async def cold_start_adapt(
         SELECT
             M.id
             , M.type, M.telegram_file_id, M.caption
-            , 'cold_start_adapt' AS recommended_by
+            , :recommended_by AS recommended_by
             , COALESCE(MS.nlikes, 0) AS nlikes
 
         FROM meme M
         INNER JOIN meme_stats MS
             ON MS.meme_id = M.id
+        INNER JOIN meme_source S
+            ON S.id = M.meme_source_id
 
         INNER JOIN user_language L
             ON L.language_code = M.language_code
@@ -530,6 +583,7 @@ async def cold_start_adapt(
             AND MS.nlikes > 1
             AND MS.nmemes_sent >= 10
             {TEXT_LIGHT_OCR_FILTER_SQL}
+            {_cold_start_guardrail_source_filter(candidate_guardrails_enabled)}
             {exclude_meme_ids_sql_filter(exclude_meme_ids)}
 
         ORDER BY -1
@@ -539,11 +593,12 @@ async def cold_start_adapt(
     """
     return await fetch_all(
         text(query),
-        _build_params(
+        _cold_start_params(
             user_id,
             limit,
             exclude_meme_ids,
-            text_light_max_ocr_words=TEXT_LIGHT_MAX_OCR_WORDS,
+            engine=COLD_START_ADAPT_RECOMMENDED_BY,
+            candidate_guardrails_enabled=candidate_guardrails_enabled,
         ),
     )
 

src/recommendations/meme_queue.py

@@ -25,7 +25,20 @@
 from src.tgbot.constants import UserType
 from src.tgbot.user_info import get_user_info
 
-COLD_START_RECOMMENDED_BY = frozenset({"cold_start_explore", "cold_start_adapt"})
+COLD_START_RECOMMENDED_BY = frozenset(
+    {
+        "cold_start_explore",
+        "cold_start_adapt",
+        "cold_start_explore_guarded",
+        "cold_start_adapt_guarded",
+    }
+)
+COLD_START_GUARDED_RECOMMENDED_BY = frozenset(
+    {
+        "cold_start_explore_guarded",
+        "cold_start_adapt_guarded",
+    }
+)
 
 
 async def get_next_meme_for_user(user_id: int) -> MemeData | None:
@@ -70,6 +83,17 @@ async def _queued_meme_is_sendable(
     meme_id: int,
     recommended_by: str | None = None,
 ) -> bool:
+    if (
+        _is_cold_start_guarded_engine(recommended_by)
+        and not settings.COLD_START_CANDIDATE_GUARDRAILS_ENABLED
+    ):
+        logging.info(
+            "discarding guarded cold_start queued meme after rollback for user_id=%s meme_id=%s",
+            user_id,
+            meme_id,
+        )
+        return False
+
     row = await fetch_one(
         text(
             """
@@ -108,6 +132,10 @@ def _is_cold_start_engine(recommended_by: str | None) -> bool:
     return recommended_by in COLD_START_RECOMMENDED_BY
 
 
+def _is_cold_start_guarded_engine(recommended_by: str | None) -> bool:
+    return recommended_by in COLD_START_GUARDED_RECOMMENDED_BY
+
+
 def _cold_start_allowed_by_realtime_state(state: dict[str, Any] | None) -> bool:
     if state is None:
         return False
@@ -288,6 +316,9 @@ async def generate_recommendations(
             meme_ids_in_queue=meme_ids_in_queue,
             random_seed=random_seed,
             cold_start_nsessions_gate_enabled=settings.COLD_START_NSESSIONS_GATE_ENABLED,
+            cold_start_candidate_guardrails_enabled=(
+                settings.COLD_START_CANDIDATE_GUARDRAILS_ENABLED
+            ),
             # FFM-1357: stop new exposure until this overlay has a CEO-owned
             # active experiment record. Existing assignment rows remain readable.
             text_light_blender_v1_enabled=False,