d-string v2

[methane]

Summary by CodeRabbit

• New Features
  • Added support for triple-quoted d strings (d, db, df, dt) including raw variants.
  • Implemented common-indent dedenting for d strings, including correct handling inside f-/t-string interpolation and format specifications.
• Bug Fixes
  • Improved validation and compatibility for string-prefix combinations involving d/D.
• Tests
  • Added comprehensive unit coverage for prefix rules, dedent/blank-line normalization, and d interactions with interpolation and format handling.

---

Note

High Risk
Large changes to core tokenizer/parser/string literal handling affect all source parsing; mistakes could cause subtle syntax or dedent bugs in f/t/d combined literals.

Overview
Adds d-strings (PEP 822): triple-quoted literals with a d prefix that strip common leading indentation from every physical line (including lines inside {...} / format specs / t-string interpolation text), normalize whitespace-only lines to a single newline, and require the body to start with a newline after the opening quotes.

The lexer recognizes d (with b, r, f, t combinations where allowed), rejects single-quoted and non-triple d forms, tracks longest common leading whitespace per active d-string, and attaches it to FSTRING_END/TSTRING_END metadata. Plain d/db/dr literals dedent in string_parser.c; df/dt defer dedent until parse end in action_helpers.c (after escape checks). u+d is rejected; Lib/tokenize.py prefix lists and prefix-discovery tests use multiline """\n""" probes.

Exports _Py_search_longest_common_leading_whitespace from unicodeobject.c for indent calculation. New Lib/test/test_dstring.py covers syntax rules, dedent edge cases, and f/t interactions.

^{Reviewed by Cursor Bugbot for commit 8700c54. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds d-string support across tokenizing, parsing, interpolation handling, and tests, including new prefix validation, dedent tracking, and dedented handling for plain, f-string, and t-string forms.

Changes

d-string language feature

Layer / File(s)	Summary
Shared dedent helper and export Objects/unicodeobject.c, Include/internal/pycore_unicodeobject.h	Renames the internal common-whitespace helper and exports it for reuse in string parsing.
Lexer prefix recognition and dedent tracking Parser/lexer/state.h, Parser/lexer/lexer.c, Parser/lexer/state.c	Adds d-prefix support, tokenizer dedent state fields, per-line indent tracking, metadata emission to end tokens, and buffer cleanup.
Plain d-string parsing Parser/string_parser.c	Parses d-prefixed strings, enforces triple-quoted newline-started forms, dedents content, and decodes the final bytes or text result.
f/t-string dedent integration Parser/action_helpers.c	Dedents debug text, replacement fields, and format specs via new helpers while deferring constant decoding for later processing.
Prefix tables and tokenizer tests Lib/tokenize.py, Lib/test/test_tokenize.py	Expands accepted prefixes to include d-string combinations and updates prefix-validation probes to use triple-quoted eval strings.
d-string test module Lib/test/test_dstring.py	Adds coverage for plain d-strings, df-string interpolation with formatting and debug output, and dt-string tuple/text capture behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is short and clearly refers to the main change: adding d-string support.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch peps/0822-d-string-v2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[methane]

@cursor review

[cursor]

Skipping Bugbot: Bugbot is disabled for this repository. Visit the Bugbot dashboard to update your settings.

[methane]

@cursor review

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

Lib/test/test_dstring.py (1)

165-177: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicate source between test_raw_dstring and test_raw_dbstring.

Both methods build the identical source and differ only in checking d() vs db(). Consider consolidating via the existing check_dbstring helper (used elsewhere in this class) to avoid the duplicated triple-quoted fixture.

♻️ Proposed consolidation

-    def test_raw_dstring(self):
-        source = r"""
-            path\\to\\file
-            keep\\n
-            """
-        self.assertEqual(d(source), "path\\to\\file\nkeep\\n\n")
-
-    def test_raw_dbstring(self):
-        source = r"""
-            path\\to\\file
-            keep\\n
-            """
-        self.assertEqual(db(source), b"path\\to\\file\nkeep\\n\n")
+    def test_raw_dbstring(self):
+        source = r"""
+            path\\to\\file
+            keep\\n
+            """
+        self.check_dbstring(source, "path\\to\\file\nkeep\\n\n")

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Lib/test/test_dstring.py` around lines 165 - 177, The two tests duplicate the
same raw triple-quoted source fixture in test_raw_dstring and test_raw_dbstring;
consolidate them using the existing check_dbstring helper in the test class.
Update the dstring test methods to share the common source setup through
check_dbstring while keeping the distinct expectations for d() and db(), so the
fixture is defined once and both behaviors are still verified.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Parser/action_helpers.c`:
- Around line 1339-1342: The early return after `PyUnicodeWriter_Create` failure
in the parser helper leaves `p->call_invalid_rules` set to 1, so restore the
previous value there just like the `error:` path and normal success flow do.
Update the `action_helpers.c` logic around `PyUnicodeWriter_Create` to reset
`p->call_invalid_rules` from `_prev_call_invalid` before returning `NULL`,
keeping the parser state consistent across recovery/second-pass handling.

---

Nitpick comments:
In `@Lib/test/test_dstring.py`:
- Around line 165-177: The two tests duplicate the same raw triple-quoted source
fixture in test_raw_dstring and test_raw_dbstring; consolidate them using the
existing check_dbstring helper in the test class. Update the dstring test
methods to share the common source setup through check_dbstring while keeping
the distinct expectations for d() and db(), so the fixture is defined once and
both behaviors are still verified.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1cdea12c-829d-4d03-bf3d-418d835f610c

📥 Commits

Reviewing files that changed from the base of the PR and between 908f438 and 8700c54.

📒 Files selected for processing (9)

• Lib/test/test_dstring.py
• Lib/test/test_tokenize.py
• Lib/tokenize.py
• Objects/unicodeobject.c
• Parser/action_helpers.c
• Parser/lexer/lexer.c
• Parser/lexer/state.c
• Parser/lexer/state.h
• Parser/string_parser.c

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8700c54. Configure here.}