Skip to content

Conversation

@dranikpg
Copy link
Contributor

@dranikpg dranikpg commented Dec 5, 2025

  • Simplfy TieredStorage::Stash logic by using OpManager::PrepareAndStash helper
  • Don't serialize SmallBin to string, return object that can be serialized to destination buffer
  • Return string_view[2] from CompactObj::GetRawString to determine size/get string copyless with small strings
  • Use Listpack in SerializedMap constructor to avoid intermediate format

@dranikpg dranikpg force-pushed the tiering-serialization-update branch from 6dee45e to 9c5e607 Compare December 5, 2025 12:25
@dranikpg dranikpg marked this pull request as ready for review December 7, 2025 09:51
@dranikpg dranikpg requested a review from romange December 7, 2025 10:36
@romange romange requested a review from Copilot December 10, 2025 13:40
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes tiering serialization by eliminating intermediate allocations and enabling zero-copy operations. The changes simplify the stash logic, improve API ergonomics, and reduce memory overhead during offloading operations.

Key Changes

  • SmallBins API refactoring: Changed FilledBin from pair<BinId, string> to a struct with an explicit SerializeBin() method that writes directly to a destination buffer, eliminating intermediate string allocations
  • Zero-copy string access: Modified CompactObj::GetRawString() to return array<string_view, 2> instead of StringOrView, enabling copyless size determination and access to small strings split across two memory segments
  • Direct listpack serialization: Updated SerializedMap::Serialize() to accept ListpackWrap directly instead of materializing an intermediate vector of string pairs

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/server/tiering/small_bins.h Redefined FilledBin as a struct with private members; added SerializeBin() method declaration; updated IsPending() to use new structure
src/server/tiering/small_bins.cc Implemented SerializeBin() to write directly to destination buffer; updated internal state management to use FilledBin struct; added memory reuse optimization for bin entries
src/server/tiering/small_bins_test.cc Changed test fixture from class to struct; added Serialize() helper method; updated all tests to use new FilledBin API
src/server/tiering/serialized_map.h Replaced SerializeSize() with EstimateSize(); changed Serialize() signature to accept ListpackWrap instead of span of pairs
src/server/tiering/serialized_map.cc Implemented EstimateSize() and updated Serialize() to iterate directly over listpack; moved kLenBytes constant to file scope
src/server/tiering/serialized_map_test.cc Added test suite setup to initialize zmalloc; updated test to create and use ListpackWrap instead of vector of pairs
src/server/tiering/op_manager.h Added PrepareAndStash() helper method declaration
src/server/tiering/op_manager.cc Implemented PrepareAndStash() to combine buffer preparation, serialization, and stashing in one call
src/server/tiering/op_manager_test.cc Refactored Stash() test helper to use new PrepareAndStash() pattern
src/server/tiered_storage.cc Simplified TryStash() to use PrepareAndStash() with bind_front for both large values and small bins; updated serialization logic to handle two-part string views
src/core/compact_object.h Changed GetRawString() return type from StringOrView to array<string_view, 2>
src/core/compact_object.cc Implemented new GetRawString() to return two views; handles both ROBJ_TAG (single view) and SMALL_TAG (two views) cases
src/core/compact_object_test.cc Updated test to handle new GetRawString() return type by combining both views
src/core/detail/listpack_wrap.h Added Bytes() method declaration
src/core/detail/listpack_wrap.cc Implemented Bytes() to return total listpack size via lpBytes()

@romange
Copy link
Collaborator

romange commented Dec 10, 2025

pls respond to copilot comments

@dranikpg
Copy link
Contributor Author

augment review

@augmentcode
Copy link

augmentcode bot commented Dec 10, 2025

Sorry, Augment does not have access to the org this pull request's branch is from.

dranikpg and others added 5 commits December 11, 2025 11:22
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>
@dranikpg dranikpg force-pushed the tiering-serialization-update branch from 7c73931 to 0a6daa2 Compare December 11, 2025 08:22
@dranikpg
Copy link
Contributor Author

@romange PTAL at the updates

return Iterator{lp_, nullptr, intbuf_};
}

size_t ListpackWrap::Bytes() const {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: BytesUsed()

explicit FilledBin(BinId id) : id{id} {
}

unsigned bytes = 0;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bytes_

}

unsigned bytes = 0;
tiering::EntryMap<std::string> entries;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

entries_

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
@dranikpg dranikpg requested a review from romange December 15, 2025 08:23
@dranikpg dranikpg merged commit 3c8f1f0 into dragonflydb:main Dec 15, 2025
10 checks passed
@dranikpg dranikpg deleted the tiering-serialization-update branch December 15, 2025 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants