[#223] List one S3 object per iRODS object in ListObjectsV2/ListObjects#232
Conversation
|
Naturally, I noticed only after opening this that the statement in my commit message about using the latest not-good replica is false. Right now, it just uses the first replica that comes back if no good replica is found. |
|
Will review once this is out of draft. |
|
Ignoring clang-format because it is - for some reason - suggesting replacing tabs with spaces even though the file is using tabs. Added a bunch of new test cases which significantly increase the amount of time it takes to run the tests. However, we know that all the possible scenarios are being covered here for at-rest data. All tests are passing. And yet... I will leave this in draft until I get around to testing with Cyberduck and S3 Browser in the Windows VM. Still haven't gotten around to that. |
|
Rebased to include fix for Minio CLI in GHA workflow |
|
Basic manual testing with S3 Browser and Cyberduck shows that things seem to be working. Gonna look into these failing 4.x tests and then will take out of draft. |
|
Fixed failing tests - it had to do with the iRODS server not recognizing itself by the Docker Compose service name hostname scheme. Ready for review. |
|
Okay, apparently, 4.3.2 tests are failing as well. Will look into those as well. I do not anticipate any changes to the C++ - just test stuff. |
korydraughn
left a comment
korydraughn
left a comment
There was a problem hiding this comment.
Looks good. Noticed a few minor things.
|
Ran tests against packages built with 4.3.2 and everything passed, so I'm not sure what happened on that last run. We shall see if things get better this time, I guess. |
|
GHA are happy. Squash if ready. |
|
Squashed |
This adds many test cases to ensure that listing an iRODS object with multiple replicas uses the expected replica information. The test cases use 3 replicas and manipulates the replica statuses, mtimes, and sizes to make sure that the expected behavior occurs in all possible cases.
…Objects This commit separates the ListObjects and ListObjectsV2 endpoints. The only meaningful difference at this time is that ListObjects will include prefixes in the Contents of the ListBucketResult returned to the client when no delimiter is specified, and ListObjectsV2 does not. This commit also makes sure that iRODS objects with multiple replicas are listed only once by ListObjectsV2 and ListObjects. This is done by examining all of the replicas and selecting the latest, marked-good one. If no good replica exists, the replica with the latest timestamp is chosen.
|
#'d, mergin |
3 participants
Addresses #223
Not my finest work, but it seems to work. Tests are passing.
Leaving in draft til I get a chance to test with Cyberduck and S3 Browser. I'd also like to add some more tests (like, retroactively) but I may punt that to another issue.