Skip to content

Fix N+1 query bottleneck in discussion list (GetPosts) - #66

Merged
boomzero merged 5 commits into
masterfrom
claude/discussion-section-performance-mlr28s
Jul 27, 2026
Merged

Fix N+1 query bottleneck in discussion list (GetPosts)#66
boomzero merged 5 commits into
masterfrom
claude/discussion-section-performance-mlr28s

Conversation

@boomzero

@boomzero boomzero commented Jul 27, 2026

Copy link
Copy Markdown
Member

Summary

The discussion list endpoint (GetPosts in Source/Process.ts) was slow because of an N+1 query pattern, not missing indexes. For every one of the 15 posts returned on a page, it made 4 additional sequential, awaited DB round trips:

  • GetTableSize on bbs_reply for the reply count
  • Select on bbs_reply for the last reply
  • Select on bbs_lock for lock status
  • Select on bbs_board for the board name

That's up to ~62 sequential round trips for a single page load. Since D1 queries go over the network per call, this latency is dominated by round-trip count, not per-query execution time — which is why adding indexes didn't help.

This PR replaces the per-post loop with a single SQL query using correlated subqueries and LEFT JOINs against bbs_board and bbs_lock, cutting the endpoint down to 2 total DB round trips (page count + page data) regardless of page size.

Also preserved a subtle existing behavior/bug: Database.Select/GetTableSize build malformed SQL when passed an empty (but defined) condition object, so the page-count query now explicitly passes undefined when no filters are set, matching the original code's behavior for the "all boards / all problems" view.

Test plan

  • Validated the new query's logic (filter combinations, zero-reply exclusion, lock join, board join) against an in-memory SQLite database mirroring the bbs_post/bbs_reply/bbs_board/bbs_lock schema from the migrations.
  • CI test suite / type check (local npm install was too slow in this sandbox to verify before pushing — deferring to CI)

Generated by Claude Code


Summary by cubic

Fixed the N+1 query in GetPosts and made the count and page queries run on the same D1 session. This drops DB round trips from ~62 to 2 per page and keeps pagination consistent.

  • Refactors
    • Use one SELECT with LEFT JOINs; compute reply_count via a correlated subquery and fetch last reply via a single ROW_NUMBER() windowed derived table JOIN.
    • Build a dynamic WHERE with bound params; keep ORDER BY, LIMIT, and OFFSET for pagination.
    • Run both COUNT and page queries via this.RawDatabase in one session to avoid snapshot mismatches.
    • Cleanups: remove dead ternary; use object instead of Object for types.

Written for commit a73e46a. Summary will update on new commits.

Review in cubic

Summary by Sourcery

Optimize the discussion list endpoint to remove an N+1 query pattern and reduce database round trips while keeping pagination and behavior consistent.

Enhancements:

  • Replace per-post database lookups in GetPosts with a single joined query that returns reply counts, last reply metadata, board names, and lock information in one pass.
  • Compute post counts and page data using shared SQL filter construction and a single RawDatabase session to maintain consistent pagination snapshots.
  • Tighten TypeScript typings by using object instead of Object and updating various response collections to be typed as arrays of objects.

GetPosts previously issued 4 extra sequential DB round trips per post
(reply count, last reply, lock status, board name) on top of the page
query, i.e. ~60+ round trips for a single 15-post page. Since D1 queries
go over the network per call, this dominates latency regardless of
indexing. Replaced the per-post loop with a single query using
correlated subqueries and joins.
@pull-request-size pull-request-size Bot added size/L and removed size/M labels Jul 27, 2026
WhereClause is guaranteed empty at the first check (nothing between
declaration and use could change it), so CodeFactor correctly flagged
it as always-true dead code.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a529fdeb80

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Source/Process.ts
The page-count query went through this.XMOJDatabase's own
D1DatabaseSession while the page data query used this.RawDatabase's
session. With D1 read replication, sequential consistency is only
guaranteed within a session, so the two reads could observe different
snapshots and corrupt pagination. Both queries now run as raw SQL
through this.RawDatabase.

Per Codex review on PR #66.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread Source/Process.ts Outdated
Replaces two separate correlated subqueries (one for last_reply_user_id,
one for last_reply_time) that each scanned bbs_reply per row, with a
single LEFT JOIN against a ROW_NUMBER()-based derived table.

Per cubic-dev-ai review on PR #66.
@sourcery-ai

sourcery-ai Bot commented Jul 27, 2026

Copy link
Copy Markdown

🧙 Sourcery has finished reviewing your pull request!


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@boomzero
boomzero merged commit 8ed1400 into master Jul 27, 2026
7 checks passed
@boomzero
boomzero deleted the claude/discussion-section-performance-mlr28s branch July 27, 2026 03:20

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The dynamically built WHERE clause and inline SQL string in GetPosts are getting fairly complex; consider extracting the query construction into a helper function or constant to improve readability and reduce the chance of subtle bugs when filters change.
  • The new RawDatabase queries access results via array indices and property names (e.g., ['results'][0]['count']); adding minimal guards or helper accessors for empty result sets and type coercion would make this pagination path more robust.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The dynamically built WHERE clause and inline SQL string in GetPosts are getting fairly complex; consider extracting the query construction into a helper function or constant to improve readability and reduce the chance of subtle bugs when filters change.
- The new RawDatabase queries access results via array indices and property names (e.g., `['results'][0]['count']`); adding minimal guards or helper accessors for empty result sets and type coercion would make this pagination path more robust.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants