Skip to content

[codex] Stream IMAP fetch literals#111

Draft
mpscholten wants to merge 1 commit into
qnikst:masterfrom
mpscholten:codex/imap-fetch-streaming
Draft

[codex] Stream IMAP fetch literals#111
mpscholten wants to merge 1 commit into
qnikst:masterfrom
mpscholten:codex/imap-fetch-streaming

Conversation

@mpscholten

@mpscholten mpscholten commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the high memory usage reported in #39 when fetching full IMAP message bodies.

This changes the IMAP fetch path so literal response bodies are read as ByteStrings directly from the stream instead of being parsed through the existing packrat/String parser and then packed back into ByteString.

Root Cause

fetch returned a ByteString, but internally it went through fetchByString, causing large IMAP literals to be represented as:

  • the full buffered server response
  • packrat parser derivations per byte
  • a Haskell String
  • a re-packed ByteString

That made full-message fetches allocate far more memory than the message size.

Changes

  • Added fetchByByteString and fetchByByteStringR.
  • Rewired fetch, fetchPeek, fetchHeader, fetchR, and related APIs through the ByteString-native path.
  • Kept fetchByString and fetchByStringR as compatibility wrappers.
  • Stream FETCH responses directly from BSStream and read {n} literals with bsGet n.
  • Added regression coverage for raw bytes and a 1 MB literal fetch.

Validation

  • nix shell github:NixOS/nixpkgs/nixos-25.05#ghc github:NixOS/nixpkgs/nixos-25.05#cabal-install -c cabal test imap-parsers
  • 21 tests passed on upstream master.

A compiled synthetic benchmark showed much lower peak RSS after streaming:

  • 2 MB literal: about 15 MB max RSS
  • 10 MB literal: about 23 MB max RSS

@mpscholten mpscholten force-pushed the codex/imap-fetch-streaming branch from fe7f0c0 to ceb90ce Compare June 5, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant