[HSTACK] - consolidate partitions if we have many small files #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ccciudatu
requested changes
Mar 12, 2025
| let mut range_start = 0; | ||
| while range_start < source_file.object_meta.size { | ||
| // Skip splitting files smaller than repartition_file_min_size | ||
| // This may result in a number of partitions slightly smaller than requested |
Collaborator
There was a problem hiding this comment.
Unless I'm misreading the code, I think this may result in a number of partitions bigger than requested, not smaller. state.1 gets increased unconditionally and then we check to see if it exceeded the size.
So we can end up with up to double the requested size in some partitions. E.g. with 100MB requested size, if state.1 is 99MB and the next file is 99MB, the partition will be 198MB.
Unless this optimisation is a game changer, I would rather keep the current behaviour, which I think is quite sane especially for large files and large limits.
aditanase
force-pushed
the
bug-repartition
branch
from
6a87a9c to
71595fe
Compare
aditanase
force-pushed
the
bug-repartition
branch
from
71595fe to
9542517
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.