Added JSON protection and selective text compression based on tenant schema

.dockerignore

@@ -3,6 +3,7 @@ __pycache__
 .pytest_cache
 .pytest-tmp
 pytest-tmp
+outputs
 .ruff_cache
 .git
 .agents

.gcloudignore

@@ -9,6 +9,7 @@
 .pytest_cache
 .pytest-tmp
 pytest-tmp
+outputs
 .ruff_cache
 .vscode
 __pycache__

CHANGELOG.md

@@ -0,0 +1,117 @@
+# Changelog
+
+This project records user-visible features, quality improvements, and deployment
+milestones over time. Historical entries before July 12, 2026 were reconstructed
+from the repository's commit history.
+
+## 2026-07-12
+
+### Added
+
+- Added JSON protection and selective text compression based on tenant schema.
+- Added tenant-authorized `<compress-json policy="...">` tagging, safe JSONPath
+  allowlists, bounded per-value compression, deterministic reconstruction, and
+  TOON-or-verbatim protection for the complete structure.
+- Added dedicated tagged JSON compression documentation and an in-app changelog.
+
+### Changed
+
+- Valid JSON objects and arrays are now protected from model compression even
+  when they are too small to benefit from TOON conversion.
+- JSON size now determines TOON eligibility rather than protection eligibility.
+
+## 2026-07-11
+
+### Changed
+
+- Improved `/compress` diagnostic results for benchmarking and attribution.
+
+## 2026-07-10
+
+### Added
+
+- Added a streamlined iframe embedding demo.
+
+## 2026-07-07
+
+### Changed
+
+- Normalized aggressiveness settings across compression paths.
+- Improved the edge compression function.
+
+## 2026-07-06
+
+### Added
+
+- Added the Cloudflare edge compression worker.
+- Added an initial GPU Docker and Cloud Run configuration.
+- Added deterministic HTML compression.
+
+## 2026-07-05
+
+### Added
+
+- Added GPU benchmark tests and recommendations.
+
+### Changed
+
+- Implemented additional deterministic compression optimizations.
+
+## 2026-06-30
+
+### Changed
+
+- Improved evaluation coverage.
+- Added deployment version and timestamp fields to `/health`.
+
+## 2026-06-27
+
+### Added
+
+- Added tenant compression settings and LoRA examples.
+
+### Changed
+
+- Improved token estimation by preferring the model tokenizer, then tiktoken,
+  then the regex fallback.
+
+## 2026-06-26
+
+### Documentation
+
+- Added the tenant adaptation plan.
+
+## 2026-06-25
+
+### Added
+
+- Added the evaluation suite and established a passing baseline.
+
+### Fixed
+
+- Fixed Docker deployment of the data directory.
+
+## 2026-06-24
+
+### Changed
+
+- Improved runtime performance.
+- Refined the HTML and JSON preprocessing pipelines.
+
+## 2026-06-23
+
+### Added
+
+- Added TOON and deterministic whitespace pipelines.
+- Added the initial test-harness application.
+- Prepared the Docker deployment path for Google Cloud Run.
+
+### Fixed
+
+- Expanded tests and fixed whitespace handling bugs.
+
+## Initial release
+
+### Added
+
+- Established the PromptCompression project and its initial API/runtime shape.

Dockerfile

@@ -26,6 +26,7 @@ RUN adduser --disabled-password --gecos "" appuser \
 COPY --chown=appuser:appuser app ./app
 COPY --chown=appuser:appuser data ./data
 COPY --chown=appuser:appuser models ./models
+COPY --chown=appuser:appuser CHANGELOG.md ./CHANGELOG.md
 
 USER appuser
 

README.md

@@ -138,6 +138,37 @@ Tenant fields are optional. They are request scoped and are not loaded from a
 local database. If `aggressiveness` is omitted, `tenant_profile.default_aggressiveness`
 is used when provided.
 
+Tagged JSON can opt into tenant-approved compression of selected long string
+values while keeping keys, types, arrays, and all other values deterministic:
+
+```json
+{
+  "tenant_profile": {
+    "json_compression_policy_id": "issue-v1",
+    "json_value_compression_paths": [
+      "$.description",
+      "$.comments[*].body"
+    ],
+    "json_value_min_tokens": 200,
+    "json_value_max_reduction": 0.25,
+    "json_value_max_values": 8
+  },
+  "text": "Review this issue:\n<compress-json policy=\"issue-v1\">{\"id\":\"ISSUE-73\",\"description\":\"Long narrative...\"}</compress-json>"
+}
+```
+
+The tag cannot authorize fields by itself: its policy must match the tenant
+profile and only allowlisted string paths are eligible. Each accepted value is
+compressed independently and JSON-escaped during reconstruction. The rebuilt
+object is then TOON-encoded when beneficial, or otherwise protected verbatim
+before any outer model-compression call. Invalid JSON, duplicate keys,
+unapproved policies, unlisted paths, and values that fail the token or maximum
+reduction gates are not model-compressed.
+
+See [Tagged JSON Compression](docs/tagged-json-compression.md) for the complete
+tag grammar, tenant schema, supported path syntax, safety gates, mode behavior,
+fallback warnings, and operational guidance.
+
 Set `include_sections` to `true` only for UI/debug views that need per-section
 labels and protected-block rendering. It defaults to `false` to keep responses
 small and skip word-label generation.

app/benchmark_ui.py

@@ -341,6 +341,7 @@
           <a class="nav-link" href="/">Compression UI</a>
           <a class="nav-link" href="/eval">Eval Suite</a>
           <a class="nav-link" href="/research">Research</a>
+          <a class="nav-link" href="/changelog">Changelog</a>
           <a class="nav-link" href="/docs">API Docs</a>
         </nav>
       </div>

app/changelog_ui.py

@@ -0,0 +1,170 @@
+from html import escape
+from pathlib import Path
+import re
+
+from app.version import DEPLOYMENT_VERSION
+
+
+CHANGELOG_PATH = Path(__file__).resolve().parents[1] / "CHANGELOG.md"
+
+
+def _inline_markdown(text: str) -> str:
+    escaped = escape(text)
+    return re.sub(r"`([^`]+)`", r"<code>\1</code>", escaped)
+
+
+def _render_changelog(markdown: str) -> str:
+    parts: list[str] = []
+    paragraphs: list[str] = []
+    list_items: list[str] = []
+
+    def flush_paragraph() -> None:
+        if paragraphs:
+            parts.append(f"<p>{_inline_markdown(' '.join(paragraphs))}</p>")
+            paragraphs.clear()
+
+    def close_list() -> None:
+        if list_items:
+            items = "".join(
+                f"<li>{_inline_markdown(item)}</li>" for item in list_items
+            )
+            parts.append(f"<ul>{items}</ul>")
+            list_items.clear()
+
+    for raw_line in markdown.splitlines():
+        line = raw_line.strip()
+        if not line:
+            flush_paragraph()
+            close_list()
+            continue
+        if line == "# Changelog":
+            continue
+        if line.startswith("## "):
+            flush_paragraph()
+            close_list()
+            parts.append(f"<h2>{_inline_markdown(line[3:])}</h2>")
+            continue
+        if line.startswith("### "):
+            flush_paragraph()
+            close_list()
+            parts.append(f"<h3>{_inline_markdown(line[4:])}</h3>")
+            continue
+        if line.startswith("- "):
+            flush_paragraph()
+            list_items.append(line[2:])
+            continue
+        if list_items and raw_line[:1].isspace():
+            list_items[-1] = f"{list_items[-1]} {line}"
+            continue
+        close_list()
+        paragraphs.append(line)
+
+    flush_paragraph()
+    close_list()
+    return "\n".join(parts)
+
+
+def _load_changelog() -> str:
+    try:
+        return CHANGELOG_PATH.read_text(encoding="utf-8")
+    except OSError:
+        return "# Changelog\n\nThe changelog is unavailable in this deployment."
+
+
+CHANGELOG_CONTENT = _render_changelog(_load_changelog())
+
+CHANGELOG_HTML = f"""<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Prompt Compression Changelog</title>
+  <style>
+    :root {{
+      color-scheme: light;
+      --bg: #f4f7fb;
+      --panel: #ffffff;
+      --text: #182230;
+      --muted: #667085;
+      --accent: #2563a6;
+      --border: #d8e0ea;
+    }}
+    * {{ box-sizing: border-box; }}
+    body {{
+      margin: 0;
+      background: var(--bg);
+      color: var(--text);
+      font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
+      line-height: 1.58;
+    }}
+    main {{ max-width: 920px; margin: 0 auto; padding: 36px 22px 72px; }}
+    header {{
+      display: flex;
+      justify-content: space-between;
+      gap: 24px;
+      align-items: flex-start;
+      margin-bottom: 24px;
+    }}
+    h1 {{ margin: 0 0 6px; font-size: clamp(2rem, 5vw, 3.2rem); line-height: 1.05; }}
+    .subhead {{ margin: 0; color: var(--muted); }}
+    .nav-links {{ display: flex; flex-wrap: wrap; justify-content: flex-end; gap: 8px; }}
+    .nav-link {{
+      border: 1px solid var(--border);
+      border-radius: 999px;
+      color: var(--accent);
+      background: var(--panel);
+      padding: 7px 11px;
+      font-size: .88rem;
+      font-weight: 700;
+      text-decoration: none;
+    }}
+    .nav-link:hover, .nav-link:focus-visible {{ border-color: var(--accent); outline: none; }}
+    article {{
+      background: var(--panel);
+      border: 1px solid var(--border);
+      border-radius: 16px;
+      box-shadow: 0 14px 34px rgba(31, 48, 76, .08);
+      padding: clamp(24px, 5vw, 48px);
+    }}
+    article > p:first-child {{ color: var(--muted); font-size: 1.05rem; }}
+    h2 {{
+      margin: 42px 0 12px;
+      padding-top: 22px;
+      border-top: 2px solid var(--border);
+      color: var(--accent);
+      font-size: 1.7rem;
+    }}
+    article h2:first-of-type {{ margin-top: 26px; }}
+    h3 {{ margin: 20px 0 8px; font-size: 1.05rem; letter-spacing: .04em; text-transform: uppercase; }}
+    p {{ margin: 0 0 14px; }}
+    ul {{ margin: 0 0 18px; padding-left: 24px; }}
+    li {{ margin: 7px 0; }}
+    code {{ border-radius: 5px; background: #edf2f7; padding: 2px 5px; font-size: .92em; }}
+    @media (max-width: 720px) {{
+      header {{ flex-direction: column; }}
+      .nav-links {{ justify-content: flex-start; }}
+    }}
+  </style>
+</head>
+<body>
+  <main>
+    <header>
+      <div>
+        <h1>Changelog</h1>
+        <p class="subhead">A dated history of product, quality, and deployment improvements. Build {DEPLOYMENT_VERSION}.</p>
+      </div>
+      <nav class="nav-links" aria-label="Primary navigation">
+        <a class="nav-link" href="/">Compression UI</a>
+        <a class="nav-link" href="/eval">Eval Suite</a>
+        <a class="nav-link" href="/benchmark">Benchmark</a>
+        <a class="nav-link" href="/research">Research</a>
+        <a class="nav-link" href="/docs">API Docs</a>
+      </nav>
+    </header>
+    <article>
+      {CHANGELOG_CONTENT}
+    </article>
+  </main>
+</body>
+</html>
+"""

app/compression_pipeline.py

@@ -433,13 +433,21 @@ def _json_segment_for_candidate(
         allow_toon: bool = True,
         leading_context: str = "",
     ) -> CompressionSegment | None:
-        if not self._is_medium_large_json(candidate):
-            return None
-
         parsed = self._parse_json(candidate)
         if parsed is None:
             return None
 
+        # Size determines whether TOON is worth attempting, not whether valid
+        # JSON is safe to expose to probabilistic model compression. Preserve
+        # small JSON verbatim as a protected structured segment.
+        if not self._is_medium_large_json(candidate):
+            return CompressionSegment(
+                text=candidate,
+                compressible=False,
+                kind="json",
+                source_text=candidate,
+            )
+
         if (
             not allow_toon
             or self._context_requires_verbatim_json(leading_context)