feat(dashboard): real-time indexing progress with WebSocket streaming#226
Conversation
- Add IndexingEventPublisher service for Redis pub/sub events
- Add /ws/repos/{repo_id}/indexing WebSocket endpoint with JWT auth
- Add /repos/{repo_id}/index/async endpoint for background indexing
- Add useRepoIndexingWebSocket hook for authenticated connections
- Add IndexingProgressModal with live file streaming UI
- Wire dashboard to show progress modal on repo add/reindex
Backend:
- services/indexing_events.py: Unified event publisher with typed events
- routes/ws_repos.py: WebSocket endpoint subscribes to indexing:{repo_id}:events
- routes/repos.py: Background task publishes progress/completion/error events
Frontend:
- hooks/useRepoIndexingWebSocket.ts: Auth-aware WebSocket with auto-reconnect
- components/IndexingProgressModal.tsx: Animated progress with file list
- components/dashboard/DashboardHome.tsx: Triggers async indexing on add
|
@DevanshuNEU is attempting to deploy a commit to the Dev's projects Team on Vercel. A member of the Team first needs to authorize it. |
📝 WalkthroughWalkthroughAdds asynchronous repository indexing with background tasks that publish Redis pub/sub events, a WebSocket endpoint to stream indexing progress (JWT auth + ownership checks), frontend WebSocket hook and modal for real-time UI, and atomic repo-status guards in repo/supabase services to prevent concurrent indexing. (39 words) Changes
Sequence DiagramsequenceDiagram
participant FE as Frontend
participant API as API Server
participant DB as Repo/DB
participant BG as Background Task
participant Redis as Redis Pub/Sub
participant WS as WebSocket Handler
FE->>API: POST /api/v1/index/async (repo_id)
activate API
API->>DB: validate ownership & try_set_indexing(repo_id)
DB-->>API: ok / already-indexing
API->>BG: schedule _run_async_indexing(repo_id)
API-->>FE: 202 + websocket URL
deactivate API
FE->>WS: WS CONNECT /ws/repos/{repo_id}/indexing?token=...
activate WS
WS->>DB: verify ownership
WS->>Redis: SUBSCRIBE indexing:{repo_id}:events
WS-->>FE: connected event
deactivate WS
activate BG
BG->>BG: clone & index repo, publish progress events
BG->>Redis: PUBLISH progress / completed / error
BG->>DB: update metadata/status
deactivate BG
Redis->>WS: deliver events to subscribers
activate WS
WS-->>FE: forward progress/completed/error events over WebSocket
deactivate WS
FE->>FE: update UI (modal progress, auto-close on completed)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Fix all issues with AI agents
In `@backend/routes/repos.py`:
- Around line 350-355: There is a TOCTOU race between the status check
(repo.get("status") == "indexing") and the later status update; implement an
atomic check-and-set in the repo manager (e.g., add
repo_manager.try_set_indexing(repo_id) that performs a single atomic update and
returns False if already "indexing") and replace the current pre-check with a
call to that method (raise the same HTTPException if it returns False); ensure
downstream code that sets status back (the existing status update path you call
later) still runs or that you clear the "indexing" flag on error so the repo
doesn't remain stuck.
In `@frontend/src/components/IndexingProgressModal.tsx`:
- Around line 84-87: The early return "if (!isOpen) return null" in the
IndexingProgressModal component prevents AnimatePresence from mounting and
therefore prevents exit animations; remove that early return and instead always
render AnimatePresence, placing the isOpen conditional inside it (e.g., render
<AnimatePresence>{isOpen && (<...modal markup...>)}</AnimatePresence>) so the
modal can animate out; update any return paths in the IndexingProgressModal
function to always return the AnimatePresence wrapper while gating the modal
children with the isOpen flag.
In `@frontend/src/hooks/useRepoIndexingWebSocket.ts`:
- Around line 147-151: The connect function and WS error handler currently set
only connection state but leave the hook's phase/error untouched, so UI gating
(hasError) never shows; update the connect logic (in useRepoIndexingWebSocket ->
connect) and the ws.onerror handler to call the phase and error setters (e.g.,
setPhase('error') and setError(new Error('missing auth token') or the ws event
error message)) in addition to setConnectionState('error'), providing a clear
descriptive message so downstream UI that reads phase/error (and hasError) will
render the error state.
- Around line 162-197: The onclose handler can act on a stale socket after
cleanup/reconnect; modify the WebSocket callbacks (at least onclose, and
similarly onmessage/onerror/onopen if desired) to first verify the event's
socket (the local ws variable or event.target) matches wsRef.current and return
early if not, so only the most-recent ws created in connect(rid) mutates state;
ensure cleanup() still clears wsRef.current and any reconnectTimeout.current so
the guard works and no stale callbacks override connectionState/error after a
new connection is established.
- Around line 205-221: The effect in useRepoIndexingWebSocket is re-running when
phase changes because phase is in the dependency array, causing connect(repoId)
to be called again when phase flips to 'completed'; remove phase from the
dependency list so the effect only reacts to repoId and session?.access_token
(keep connect and cleanup in deps or ensure they are stable via useCallback),
and if your linter complains add an explicit comment disabling exhaustive-deps
for this effect with a short rationale that phase should not trigger reconnects;
update references to
setConnectionState/setPhase/setProgress/setRecentFiles/setError remain inside
the cleanup branch as-is.
- Around line 135-139: The error branch for the websocket 'error' event only
reads data.message and falls back to 'Unknown error', so if the server supplies
data.error it is ignored; update the 'error' case (inside
useRepoIndexingWebSocket) to prefer data.error over data.message (e.g., use
data.error || data.message || 'Unknown error') and propagate that value to
setError, setIsRecoverable, and onErrorRef.current so the UI receives the
server-provided error text and recoverable flag.
🧹 Nitpick comments (3)
frontend/src/components/dashboard/DashboardHome.tsx (1)
113-116: Consider using completion stats in the toast.The
onCompletedcallback provides(repoId, stats)with indexing statistics, buthandleIndexingCompleteignores them. You could enhance the success toast with stats like function count.♻️ Optional enhancement
- const handleIndexingComplete = async () => { + const handleIndexingComplete = async (_repoId: string, stats?: { functions_indexed: number }) => { await fetchRepos() - toast.success('Indexing complete!', { description: `${indexingRepoName} is ready for search` }) + toast.success('Indexing complete!', { + description: stats + ? `${indexingRepoName} indexed ${stats.functions_indexed} functions` + : `${indexingRepoName} is ready for search` + }) }backend/routes/ws_repos.py (1)
128-131: Useasyncio.get_running_loop().time()instead of deprecated API.
asyncio.get_event_loop()is deprecated in newer Python versions when called from a coroutine. Useasyncio.get_running_loop()ortime.monotonic()for timing.♻️ Proposed fix
- last_activity = asyncio.get_event_loop().time() + last_activity = asyncio.get_running_loop().time() while True: - current_time = asyncio.get_event_loop().time() + current_time = asyncio.get_running_loop().time()backend/routes/repos.py (1)
397-402: Consider returning HTTP 202 Accepted for async operation.The endpoint returns 200 OK, but 202 Accepted is more semantically correct for async operations that haven't completed yet.
♻️ Proposed fix
+from fastapi.responses import JSONResponse - return { + return JSONResponse( + status_code=202, + content={ "status": "indexing", "repo_id": repo_id, "message": "Indexing started. Connect to WebSocket for progress.", "websocket_url": f"/api/v1/ws/repos/{repo_id}/indexing" - } + } + )
…TOU race - Add try_set_indexing_status() to supabase_service using .neq() filter - Add try_set_indexing() wrapper to repo_manager - Replace non-atomic status check in /index/async with atomic method - Returns 409 if repo already indexing, prevents duplicate indexing jobs - Error handler already sets status to 'error' so repo won't stay stuck
- Remove early return 'if (!isOpen) return null' which prevented exit animations
- Always render AnimatePresence wrapper
- Gate modal content with {isOpen && (...)} inside AnimatePresence
- Now modal properly animates out when closing
- No auth token: set phase='error', error='Authentication required...' - ws.onerror: set phase='error', error='WebSocket connection error' - Max reconnects: set phase='error' (error was already set) - Catch block: set phase='error', error=err.message - UI components using hasError will now properly show error state
…nnect - Add 'if (ws !== wsRef.current) return' guard to onopen, onmessage, onerror, onclose - Prevents old socket events from mutating state after a new connection is created - cleanup() clears wsRef.current so guard works correctly
…ompletion - Add phaseRef to track phase without causing effect re-runs - Use phaseRef.current inside effect instead of phase state - Remove phase from dependency array with eslint-disable comment - Prevents connect(repoId) being called when phase changes to 'completed'
- Error message now uses: data.error || data.message || 'Unknown error' - Server-provided error code is now visible to UI - Same error message passed to setError, setIsRecoverable, and onErrorRef
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Fix all issues with AI agents
In `@backend/routes/repos.py`:
- Around line 329-401: The route index_repository_async currently returns 200 by
default even though the docstring says it should return 202; update the FastAPI
route to send a 202 status by either adding status_code=202 to the router.post
decorator for index_repository_async or explicitly returning a
Response/JSONResponse with status_code=202 while leaving the existing payload
and BackgroundTasks scheduling (and keeping the call to _run_async_indexing and
repo_manager.try_set_indexing unchanged) so clients receive the documented 202
Accepted.
- Around line 287-304: index_repository_with_progress currently returns a
function count (total_functions) but the code uses that value for file_count and
IndexingStats.files_processed; modify the indexing flow to also capture the
total_files value provided by the progress callback (or the incremental-progress
variant) and use that for repo_manager.update_file_count(repo_id, total_files)
and IndexingStats.files_processed=total_files while keeping
functions_indexed=total_functions; apply the same change in the incremental
indexing path so metrics.record_indexing(repo_id, duration, total_functions)
remains unchanged but file count and the publisher.publish_completed
IndexingStats use total_files instead of total_functions.
In `@frontend/src/components/IndexingProgressModal.tsx`:
- Around line 57-82: The auto-close timeout stored in closeTimeoutRef isn't
cleared when the user manually closes or retries (handleClose, handleRetry) or
when isOpen becomes false, which can cause stale callbacks to fire; update
handleClose and handleRetry to clear and nullify closeTimeoutRef (using
clearTimeout on closeTimeoutRef.current) before calling
reset()/onClose()/onRetry(), and extend the existing useEffect cleanup or add an
effect watching isOpen to also clear and nullify closeTimeoutRef when isOpen
becomes false so no stale timeout can trigger later.
In `@frontend/src/hooks/useRepoIndexingWebSocket.ts`:
- Around line 138-145: The switch case for 'error' in useRepoIndexingWebSocket
leaks the local variable errorMessage across cases; wrap the entire case 'error'
body in a block to scope errorMessage locally and satisfy Biome. Locate the case
'error' block (the code that calls setPhase('error'), computes errorMessage,
calls setError, setIsRecoverable, and onErrorRef.current) and enclose its
statements within braces { ... } so errorMessage is block-scoped while
preserving existing behavior.
Vite strips TypeScript types at runtime, so IndexingPhase and IndexingStats must be imported with 'import type' to avoid runtime errors.
Added status_code=202 to router.post decorator as documented in docstring.
- Add tracked_total_files captured from progress callback via nonlocal - Use total_files for repo_manager.update_file_count() and IndexingStats.files_processed - Keep total_functions for functions_indexed and metrics.record_indexing() - For incremental: use repo's existing file_count or re-analyze if missing
- Clear and nullify closeTimeoutRef in handleClose() - Clear and nullify closeTimeoutRef in handleRetry() - Add effect to clear timeout when isOpen becomes false - Prevents stale callbacks from firing after manual interaction
Wrap case 'error' body in braces to scope const errorMessage locally. Satisfies Biome linter and prevents variable leakage across cases.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@backend/routes/repos.py`:
- Around line 347-411: Update the docstring for the route that triggers async
indexing (the function that returns the indexing status and websocket_url) to
use the full versioned WebSocket path "/api/v1/ws/repos/{repo_id}/indexing"
instead of "/ws/repos/{repo_id}/indexing" so it matches the websocket_url
returned and the registered endpoint; locate the docstring at the top of the
async indexing handler in routes (the same function that calls get_repo_or_404,
repo_validator.analyze_repo, repo_manager.try_set_indexing, and
background_tasks.add_task) and replace the two-line path text accordingly.
In `@frontend/src/hooks/useRepoIndexingWebSocket.ts`:
- Around line 232-259: The reconnectAttempts counter isn't being cleared for new
sessions which can block future retries; update the effect that runs on
repoId/session change (the useEffect that calls connect and cleanup) to reset
reconnectAttempts (e.g., reconnectAttemptsRef.current = 0) when starting a fresh
session, and also reset it inside the reset callback (the reset function) so
manual resets also clear reconnectAttempts; reference the reconnectAttempts
ref/variable, the useEffect that calls connect(repoId), and the reset function
to locate where to set it to zero.
- Reset reconnectAttempts.current = 0 when starting fresh connection in useEffect - Reset reconnectAttempts.current = 0 in reset() callback - Ensures future retries aren't blocked by stale counter from previous sessions
Update docstring to use full versioned path /api/v1/ws/repos/{repo_id}/indexing
to match the actual registered endpoint and websocket_url response.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@frontend/src/hooks/useRepoIndexingWebSocket.ts`:
- Around line 111-119: In the 'progress' case inside
useRepoIndexingWebSocket.ts, compute a fallback percent when data.percent is
missing by deriving it from data.files_processed and data.files_total (e.g.,
percent = files_total > 0 ? Math.round((files_processed / files_total) * 100) :
0), guard against division by zero and clamp to 0–100, then call setProgress
with percent: data.percent ?? computedPercent (and keep the other fields as
currently set); update references in the same case handling where setPhase and
setProgress are called.
0.2s was too short - client might not process message before close. Increased to 1.0s to ensure reliable delivery of terminal events.
Give users more time to see completion stats before modal closes.
Previously, repos were added to DB before size check, leaving orphan 'pending' entries when rejected. Now we: - Delete repo if analysis fails (return 500) - Delete repo if size check fails (return 403) - No more pending cards for rejected repos
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/routes/repos.py (1)
67-146: Preserve intended 403/500 responses by re‑raising HTTPException.The new analysis/size-check failures raise HTTPException inside the try block, but the broad
except Exceptionat the end converts them to 400. This changes the documented semantics and hides 403/500 from clients.🐛 Suggested fix
try: # ... existing logic ... return { "repo_id": repo["id"], "status": "added", "indexing_blocked": False, "analysis": analysis.to_dict(), "message": "Repository added successfully. Ready for indexing." } + except HTTPException: + raise except Exception as e: logger.error("Failed to add repository", error=str(e), user_id=user_id) capture_exception(e) raise HTTPException(status_code=400, detail=str(e))
🤖 Fix all issues with AI agents
In `@backend/routes/repos.py`:
- Around line 319-329: The publish_completed call is passing repo_id twice;
instead pass the current user_id as the first argument so the event is scoped to
the user. Update the call to publisher.publish_completed(user_id, repo_id,
IndexingStats(...)) in the block that publishes the completion event
(references: publish_completed, IndexingStats, repo_id, user_id) so the method
signature entity_id, repo_id, stats is respected.
🧹 Nitpick comments (1)
frontend/src/components/IndexingProgressModal.tsx (1)
103-132: Add basic dialog ARIA attributes for accessibility.This is a modal dialog; adding
role="dialog",aria-modal="true", and a label improves screen‑reader UX.♿ Suggested change
- <div className="flex items-center justify-between px-6 py-4 border-b border-zinc-800"> - <h3 className="text-lg font-semibold text-white"> + <div className="flex items-center justify-between px-6 py-4 border-b border-zinc-800"> + <h3 id="indexing-progress-title" className="text-lg font-semibold text-white"> Indexing {repoName} </h3> <button onClick={handleClose} className="p-1 text-zinc-400 hover:text-white transition-colors rounded-lg hover:bg-zinc-800" > <X className="w-5 h-5" /> </button> </div>- <motion.div + <motion.div initial={{ scale: 0.95, opacity: 0 }} animate={{ scale: 1, opacity: 1 }} exit={{ scale: 0.95, opacity: 0 }} transition={{ type: 'spring', damping: 25, stiffness: 300 }} className="relative w-full max-w-lg mx-4 bg-[`#0d0d14`] border border-zinc-800 rounded-2xl shadow-2xl overflow-hidden" onClick={e => e.stopPropagation()} + role="dialog" + aria-modal="true" + aria-labelledby="indexing-progress-title" >
Redis pub/sub doesn't buffer - events sent before subscription are lost. Added 1.5s delay + initial progress event to confirm connection working. Also improved progress bar animation easing.
Root cause: Progress was only reported during file extraction (~2s). Embedding generation takes ~35s with no progress updates. Fix: - Added functions_total to progress model - Percent now weighted: 20% file extraction, 80% embedding - Reports progress during embedding phase with function count - UI will now show gradual 0-100% over full indexing duration
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@frontend/src/components/IndexingProgressModal.tsx`:
- Around line 165-179: The list rendering for recentFiles uses key={file} which
can be non-unique and cause React render/animation glitches; update the map call
that renders the motion.div (the recentFiles.slice(0, 5).map callback) to use a
stable unique key per item such as a composite of file + index or a unique event
id if available (e.g., `${file}-${i}` or use an id property from the recentFiles
item) so that the motion.div elements and their animations are keyed uniquely.
- Around line 113-131: The modal rendered in IndexingProgressModal is missing
dialog semantics and keyboard affordances; add role="dialog" and
aria-modal="true" to the motion.div, give the title element (the h3 showing
"Indexing {repoName}") a unique id and reference it via aria-labelledby on the
dialog, add an accessible label to the icon-only close button (e.g.,
aria-label="Close dialog" on the button that calls handleClose), and implement
focus management and Escape handling by focusing the close button (or dialog)
when opened and attaching a keydown handler that calls handleClose on Escape
(and ensure you clean up the listener on unmount). Ensure existing onClick={e =>
e.stopPropagation()} remains.
🧹 Nitpick comments (3)
backend/services/indexing_events.py (2)
38-48: Potential division by zero whenfunctions_total > 0butfiles_total == 0.If
functions_total > 0andfiles_total == 0, the condition on line 41 evaluates toFalse(due to short-circuit), falling through to line 45 wherefiles_total > 0is alsoFalse. This is safe. However, if in the future someone reorders conditions or the logic changes, consider adding an explicit guard.The current logic is correct but could be clearer:
♻️ Optional: explicit guard for clarity
def __post_init__(self): # If we have functions_total, we're in embedding phase (slow) - weight it 80% # File extraction is fast, weight it 20% + if self.files_total <= 0: + self.percent = 0 + return + - if self.functions_total > 0 and self.files_total > 0: + if self.functions_total > 0: file_progress = (self.files_processed / self.files_total) * 20 # 0-20% embed_progress = (self.functions_found / self.functions_total) * 80 # 0-80% self.percent = int(file_progress + embed_progress) - elif self.files_total > 0: + else: # Still in file extraction phase (0-20%) self.percent = int((self.files_processed / self.files_total) * 20)
86-91: Channel name truncation may hide debugging info.The channel is truncated to 40 characters in the log (line 88). With the format
indexing:{entity_id}:events, ifentity_idis a UUID (36 chars), the full channel is ~50 chars. The truncation cuts off part of the entity ID, which could hinder debugging.♻️ Suggested fix
logger.info( "Published event to Redis", - channel=channel[:40], + channel=channel, event_type=event.get("type"), subscribers=result )backend/routes/repos.py (1)
258-260: Document the workaround with a TODO or link to issue.The 1.5s sleep is a necessary workaround for Redis pub/sub not buffering messages, but this coupling between producer and consumer timing is fragile. Consider adding a TODO comment or linking to an issue tracking a more robust solution (e.g., hybrid approach with initial state fetch + pub/sub for updates).
♻️ Suggested documentation
# Wait for WebSocket client to connect and subscribe # Redis pub/sub doesn't buffer - events sent before subscription are lost + # TODO: Consider Redis Streams or initial state fetch to avoid timing dependency await asyncio.sleep(1.5)
Added role='dialog', aria-modal='true', and aria-labelledby for screen reader accessibility.
Backend (indexing_events.py): - Add explicit guard for files_total <= 0 (division by zero) - Remove channel truncation in logs for better debugging Backend (repos.py): - Add TODO comment for Redis Streams alternative Frontend (IndexingProgressModal.tsx): - Use composite key for recentFiles list (file + index) - Add aria-label to close button - Add Escape key handler to close modal
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/routes/repos.py (1)
67-146: Preserve HTTPException status codes.
The new analysis/size checks raise HTTPException inside the try block, but the broadexcept Exceptionconverts them to 400, losing 403/500 and structured details. Add an explicitexcept HTTPException: raise.🛠️ Proposed fix
try: # Clone repo first user_id_hash = hashlib.sha256(user_id.encode()).hexdigest() repo = repo_manager.add_repo( name=request.name, git_url=request.git_url, branch=request.branch, user_id=user_id, api_key_hash=user_id_hash ) @@ return { "repo_id": repo["id"], "status": "added", "indexing_blocked": False, "analysis": analysis.to_dict(), "message": "Repository added successfully. Ready for indexing." } + except HTTPException: + raise except Exception as e: logger.error("Failed to add repository", error=str(e), user_id=user_id) capture_exception(e) raise HTTPException(status_code=400, detail=str(e))
🤖 Fix all issues with AI agents
In `@backend/services/indexing_events.py`:
- Around line 38-52: In __post_init__ of the indexing event dataclass, clamp the
computed self.percent to the 0–100 range before assigning it so any overshoot or
negative values are bounded; after computing percent in both branches (the
file+embed branch that calculates file_progress + embed_progress and the
file-only branch), wrap the final value with a clamp (e.g., max(0, min(100,
computed_value))) and assign that to self.percent to ensure the UI never sees
values <0 or >100.
Defensive programming to prevent UI showing values like '107%' if counts ever overshoot due to race conditions or off-by-one bugs.
- Add AnimatePresence for smooth enter/exit transitions - Spring physics for natural feeling motion - Staggered delays for cascade effect - Current file highlighted (brighter text + icon) - Subtle scale animation on enter/exit - Layout animation for smooth reordering
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Real-time Indexing Progress
Replaces the "spinner → wait → manual refresh" experience with live WebSocket streaming that shows files being indexed in real-time.
Before
After
Changes
Backend
services/indexing_events.pyroutes/ws_repos.pyindexing:{repo_id}:eventsroutes/repos.pyPOST /repos/{id}/index/asyncendpoint + background task with progress callbacksmain.py/ws/repos/{repo_id}/indexingFrontend
hooks/useRepoIndexingWebSocket.tscomponents/IndexingProgressModal.tsxcomponents/dashboard/DashboardHome.tsxArchitecture
Modal Features
Event Types
Testing
Also works for re-indexing existing repos via the Overview tab.
Summary by CodeRabbit
New Features
Bug Fixes
✏️ Tip: You can customize this high-level summary in your review settings.