Optimize oplog size by stripping locations of large objects#2715
Optimize oplog size by stripping locations of large objects#2715delthas wants to merge 5 commits intodevelopment/9.3from
Conversation
Issue: BB-491 Signed-off-by: Thomas Flament <thomas.flament@scality.com>
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files
... and 1 file with indirect coverage changes
@@ Coverage Diff @@
## development/9.3 #2715 +/- ##
===================================================
- Coverage 74.57% 74.29% -0.29%
===================================================
Files 200 200
Lines 13562 13592 +30
===================================================
- Hits 10114 10098 -16
- Misses 3438 3484 +46
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
|
Manually tested with varying values and single-part, empty, and multi-part objects, works. Also manually tested changing the configuration on the fly, properly applies, and properly resumes the pipeline with the token. |
delthas
force-pushed
the
improvement/BB-491/strip-oplog-location
branch
from
d48fd8a to
e34f3dc
Compare
delthas
force-pushed
the
improvement/BB-491/strip-oplog-location
branch
from
89e94a0 to
d4ad94a
Compare
delthas
force-pushed
the
improvement/BB-491/strip-oplog-location
branch
from
d4ad94a to
db7640b
Compare
|
Tested on a live cluster, working. |
| 'fullDocument.value.location': { | ||
| $cond: { | ||
| if: { $gte: [ | ||
| '$fullDocument.value.content-length', |
There was a problem hiding this comment.
how does $gte behave if there is no $fullDocument.value.content-length field : does it go into the then or else case? (i.e. how does it compare undefined vs a number)
may be good to add a test for this specific case (not sure it really happen, but maybe in some corner case like pushing empty objet...), to ensure we are not adding any risk
There was a problem hiding this comment.
In MongoDB's BSON comparison order, null/missing values are less than any number (MinKey < Null < Numbers < ...). So when content-length is missing, $gte: [null, threshold] evaluates to false, and the else branch is taken, preserving the location field as-is.
This is the correct behavior — if we don't know the object size, we keep the location.
Working on a test.
There was a problem hiding this comment.
Actually we do not have functional tests using Mongo + Kafka + Kafka Connect here? So not really testable as is here. Maybe in Zenko?
There was a problem hiding this comment.
I was thinking more of a "unit" test, i.e. just trying to use the pipeline returned by getPipeline with mongo : create object in mongo, then make a query with the pipeline and check the result (and repeat with different "kinds" of objects)
There was a problem hiding this comment.
if later you can add functional tests in zenko i think it would be nice
4 participants
Issue: BB-491