Bound memory by writing ciphertext and plaintext in chunks - #22
Merged
Conversation
`__main__` built the entire ciphertext with `bytes(encrypter(...))` and the
entire plaintext with `bytes(decrypt(...))` before writing either, so peak memory
scaled with the file rather than staying constant. Encrypting two 4 MiB
plaintexts into an 11 MiB ciphertext:
master chunked
encrypt peak 75.4 MB 42.1 MB -44%
decrypt peak 36.5 MB 29.9 MB -18%
The saving grows with input size, since what is no longer held is proportional to
the ciphertext.
`Encrypter.blocks()` yields whole blocks -- the natural unit here, one per encoded
nibble-gram -- and `chunks()` batches those to roughly 64 KiB. `decrypt_chunks()`
does the same coming back. `__iter__` still flattens to individual ints and
`bytes(encrypter)` still works, so the existing library API is unchanged; the
encrypter headers now yield `bytes` rather than ints, which is the one visible
difference for anything calling `get_header()` directly.
Separately, a real if modest speedup: `pack_grams` was called twice per emitted
block, once to test membership and again to look up the value. `consume_grams`
now packs once per position and passes the key down, so `can_encode` and
`encode_block` take the packed key instead of the gram tuple. Encrypting 1 MiB
goes from 7.03s to 6.22s, and 256 KiB from 1.83s to 1.52s.
Worth being clear about what this is *not*: profiling showed the per-byte `yield`
overhead I expected to find is not where encryption spends its time. At 256 KiB
of plaintext the cost is the nibble walk itself -- 3.9M `peek_nibbles` calls, 1.2M
`_grams_at` calls -- and that is irreducibly per-nibble Python. Encryption remains
around 6 seconds per MiB. This commit buys bounded memory and removes duplicated
work; it does not make the walk fast.
11 new tests check that chunked output reassembles byte-for-byte into what the
whole-buffer paths produce, across chunk sizes from 1 byte to 1 MiB, and that
chunk sizes track the requested granularity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01367hFob9sd4xpDmVT4uoFy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
__main__built the entire ciphertext withbytes(encrypter(...))and the entire plaintext withbytes(decrypt(...))before writing either, so peak memory scaled with the file rather than staying constant. Encrypting two 4 MiB plaintexts into an 11 MiB ciphertext:The saving grows with input size, since what is no longer held is proportional to the ciphertext.
API
Encrypter.blocks()yields whole blocks — the natural unit here, one per encoded nibble-gram — andchunks()batches those to roughly 64 KiB.decrypt_chunks()does the same coming back.__iter__still flattens to individual ints andbytes(encrypter)still works, so the existing library API is unchanged. The one visible difference: encrypter headers now yieldbytesrather than ints, which affects anything callingget_header()directly.A real if modest speedup
pack_gramswas called twice per emitted block — once to test membership, again to look up the value.consume_gramsnow packs once per position and passes the key down, socan_encodeandencode_blocktake the packed key instead of the gram tuple.What this is not
Worth being clear: profiling showed the per-byte
yieldoverhead I expected to find is not where encryption spends its time. At 256 KiB of plaintext the cost is the nibble walk itself — 3.9Mpeek_nibblescalls, 1.2M_grams_atcalls — and that is irreducibly per-nibble Python. Encryption remains around 6 seconds per MiB.This PR buys bounded memory and removes duplicated work. It does not make the walk fast. If encryption throughput matters later, that walk is the thing to attack, and it would be a much more invasive change.
Verification
11 new tests check that chunked output reassembles byte-for-byte into what the whole-buffer paths produce, across chunk sizes from 1 byte to 1 MiB, and that chunk sizes track the requested granularity. 234 passed.
🤖 Generated with Claude Code
https://claude.ai/code/session_01367hFob9sd4xpDmVT4uoFy