Is zipROFS slow?

[christophgil]

ZipROFS is exactly what we need.

We have tons of mass spectrometry data, each record is a Zip containing several files.
However, they are very large, some have several Gigabytes.

I measure the time for reading a file of 600MB
time nocache cat my-file >/dev/null

Mounting single zip using fuse-zip, this takes 3 sec, ZipROFS takes 6 min.

Is there a trick to make ZipROFS faster?

I tried blksize=8096, but this does not have any impact.

I downloaded the latest ZipROFS.
I installed fusepy
pip install fusepy -U
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: fusepy in ./.local/lib/python3.10/site-packages (3.0.1)

Many thanks for this nice software.

[JuniorJPDJ]

Try also ratarmount

[qu1ck]

I wrote this with "lots of zip files with lots of small files" use case in mind and it performs good enough for me. I haven't run any benchmarks on large files, but a difference of 2 orders of magnitude does not seem right.

I'll investigate.

[qu1ck]

I tested on a zip file with 2 random files 512MB each. dd can achieve 30MB/s reads

$ ls -lah
total 0
dr--r--r-- 1 quick quick 1.1G Jan 31 12:10 .
drwxr-xr-x 2 quick quick 4.0K Jan 31 12:10 ..
-r-xr-xr-x 1 quick quick 512M Jan 31 12:09 random2.data
-r-xr-xr-x 1 quick quick 512M Jan 31 12:08 random.data

$ dd if=./random2.data  of=/dev/null bs=1M status=progress
510656512 bytes (511 MB, 487 MiB) copied, 16 s, 31.9 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 16.8555 s, 31.9 MB/s

$ nocache dd if=./random2.data  of=/dev/null bs=1M status=progress
507510784 bytes (508 MB, 484 MiB) copied, 16 s, 31.7 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 17.0205 s, 31.5 MB/s

cat is significantly slower on the same file at ~3MB/s average. It starts at ~10 and by the end of file slows to a crawl of < 1MB/s.

Looking at debug logs, reason becomes obvious: cat does out of order reads somehow. Instead of reading each chunk strictly one after the other like dd it sometimes reads ahead and then seeks back to previous chunk and jumps around like that.
This destroys zip read performance as each seek needs full file read from the start, that's just how zip works.

It's possible that it's not cat's "fault" but there is some reordering of syscalls happening in fuse or fusepy or python itself is not handling the calls in order.

Either way I suggest using a program with strong sequence guarantees like dd to copy the file to a ramdisk or more suitable location before you use it in something that needs fast random access performance. 30MB/s is not particularly fast but it's a lot better than 3MB/s.

Better performance for sequential reads would need a reimplementation in C/C++/Rust.

[christophgil]

Cool, thank you.

…

On Tue, Jan 31, 2023 at 6:39 PM qu1ck ***@***.***> wrote: I wrote this with "lots of zip files with lots of small files" use case in mind and it performs good enough for me. I haven't run any benchmarks on large files, but a difference of 2 orders of magnitude does not seem right. I'll investigate. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRZU6AYDWPM2A4WJCIVAMTWVFL5RANCNFSM6AAAAAAUMLVTNU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[christophgil]

You are right, I tried checksum computation and the performance is much better. 14 sec vs 3 sec for ZipROFS and fuse-zip respectively. This would be sufficient. I need to check how fast our programs read the data and maybe ZipROFS is the way to go. rhash --printf='%C' --crc32 I wonder whether one could increase blocksize or tune with other parameters.

…

On Tue, Jan 31, 2023 at 9:09 PM qu1ck ***@***.***> wrote: I tested on a zip file with 2 random files 512MB each. dd can achieve 30MB/s reads $ ls -lah total 0 dr--r--r-- 1 quick quick 1.1G Jan 31 12:10 . drwxr-xr-x 2 quick quick 4.0K Jan 31 12:10 .. -r-xr-xr-x 1 quick quick 512M Jan 31 12:09 random2.data -r-xr-xr-x 1 quick quick 512M Jan 31 12:08 random.data $ dd if=./random2.data of=/dev/null bs=1M status=progress 510656512 bytes (511 MB, 487 MiB) copied, 16 s, 31.9 MB/s 512+0 records in 512+0 records out 536870912 bytes (537 MB, 512 MiB) copied, 16.8555 s, 31.9 MB/s $ nocache dd if=./random2.data of=/dev/null bs=1M status=progress 507510784 bytes (508 MB, 484 MiB) copied, 16 s, 31.7 MB/s 512+0 records in 512+0 records out 536870912 bytes (537 MB, 512 MiB) copied, 17.0205 s, 31.5 MB/s cat is significantly slower on the same file at ~3MB/s average. It starts at ~10 and by the end of file slows to a crawl of < 1MB/s. Looking at debug logs, reason becomes obvious: cat does out of order reads somehow. Instead of reading each chunk strictly one after the other like dd it sometimes reads ahead and then seeks back to previous chunk and jumps around like that. This destroys zip read performance as each seek needs full file read from the start, that's just how zip works. It's possible that it's not cat's "fault" but there is some reordering of syscalls happening in fuse or fusepy or python itself is not handling the calls in order. Either way I suggest using a program with strong sequence guarantees like dd to copy the file to a ramdisk or more suitable location before you use it in something that needs fast random access performance. 30MB/s is not particularly fast but it's a lot better than 3MB/s. Better performance for sequential reads would need a reimplementation in C/C++/Rust. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRZU6FBJURTXYVSWY4D7PDWVF5P3ANCNFSM6AAAAAAUMLVTNU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[qu1ck]

I found that it was actually FUSE that was reordering reads because async reads are enabled by default when they are supported by kernel.

It took some monkeypatching to disable this since fusepy does not support passing requested capability flags at the moment.
But with latest commit I was able to bring cat performance to the same level, I get ~28MB/s now.

Async reads can still be enabled by passing async option to ziprofs.

Try the latest version.

[JuniorJPDJ]

You may need my zipfile patches applied when accessing uncompressed zip files. Those make random reads muuuch better: python/cpython#27737

[qu1ck]

That will help too but looking at underlying lib sources there is just too much memory copying going around to achieve close to native speeds. I found at least 2 places where read buffer is copied (once in ZipFileExt, once in fusepy).

To get somewhat close to native read speeds this needs to be reimplemented in a language that is not afraid of pointers.

[christophgil]

Yes indeed, the biggest file in our zips is uncompressed. Thanks JunioJPDJ - I will download your version soon. A first test indicates that it works with our software which is a Windows exe using closed source dlls for file loading. It runs via Wine. It accepts folders ending with ".d". I solved this by creating empty folders and symlinking the required files into it. I tried to understand the python code. I want to change it such that files ending with ".d.Zip" would give rise to folders ending with .d but could not yet find out how. I haven't done Python for 20 years, however your code looks beautiful and understandable. Thank you for this great file system which will make things so much easier for us.

…

On Wed, Feb 1, 2023 at 4:45 AM qu1ck ***@***.***> wrote: That will help too but looking at underlying lib sources there is just too much memory copying going around to achieve close to native speeds. I found at least 2 places where read buffer is copied (once in ZipFileExt, once in fusepy). To get somewhat close to native read speeds this needs to be reimplemented in a language that is not afraid of pointers. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRZU6FBSGTQ6AUCEDCNIIDWVHS5VANCNFSM6AAAAAAUMLVTNU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[qu1ck]

Changing what name folder corresponding to zip files will have in mounted tree is not trivial. Assumption that any path containing *.zip as one of it's element is actually a path within a zip and not a real path in underlying fs is baked in pretty deep.

If you want to change that you need to look at get_zip_path() function. Currently it returns a prefix of the path to zip file if it's a path within a zip file and None otherwise.

You would need to change the logic of get_zip_path() and every caller of it because previously zip file path was a strict prefix but with the modified paths it will not be, example: "/path/to/some.d/file.txt" may actually be a file.txt inside "/path/to/some.d.zip".

[christophgil]

Thank you - this is what I was already assuming. Either I leave it the way it is and live with symlink creation or I will make a magic prefix like my_file.d.Zip -> ZiP_my_file.d I would take ZiP_.*\.d as a recognition pattern. This would work - wouldn't it? Cheers C

…

On Fri, Feb 3, 2023 at 12:25 AM qu1ck ***@***.***> wrote: Changing what name folder corresponding to zip files will have in mounted tree is not trivial. Assumption that any path containing *.zip as one of it's element is actually a path within a zip and not a real path in underlying fs is baked in pretty deep. If you want to change that you need to look at get_zip_path() function. Currently it returns a prefix of the path to zip file if it's a path within a zip file and None otherwise. You would need to change the logic of get_zip_path() and every caller of it because previously zip file path was a strict prefix but with the modified paths it will not be, example: "/path/to/some.d/file.txt" may actually be a file.txt inside "/path/to/some.d.zip". — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRZU6GNTXG27GXVJA6CL53WVRGA5ANCNFSM6AAAAAAUMLVTNU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[qu1ck]

Yes, it would but complexity is about the same. You still have to add translation logic from real path to virtual path and vice versa.

Maybe what would be easier for your case is to drop the .zip extension from your files and just name your archives as whatever.d.
As long as they are still in zip format I think it would be a one line change in ZipROFS to treat them as zip, just change the condition here
https://github.com/openscopeproject/ZipROFS/blob/dev/ziprofs.py#L110

to be if path[-2:] == ".d" and ...

[christophgil]

Seems that patching was successful It is now possible to alter the folder names. Implementation: Starting from the fork JuniorJPDJ <https://github.com/JuniorJPDJ> / ZipROFS <https://github.com/JuniorJPDJ/ZipROFS>, I introduced two methods which can be customized to user's needs. def zipfilename_real_to_virtual(name: str): def zippath_virtual_to_real(path: str): After testing on real data, I will upload my version. Thanks so much for your great help.

…

On Fri, Feb 3, 2023 at 9:43 AM qu1ck ***@***.***> wrote: Yes, it would but complexity is about the same. You still have to add translation logic from real path to virtual path and vice versa. Maybe what would be easier for your case is to drop the .zip extension from your files and just name your archives as whatever.d. As long as they are still in zip format I think it would be a one line change in ZipROFS to treat them as zip, just change the condition here https://github.com/openscopeproject/ZipROFS/blob/dev/ziprofs.py#L110 to be if path[-2:] == ".d" and ... — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRZU6COCKNW47LICYIOLZTWVTHNBANCNFSM6AAAAAAUMLVTNU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JuniorJPDJ]

Don't use my fork of ZipROFS - it is out of date and merged to branches here.
Try using python 3.12 or get zipfile.py library from it and copy near your software.

[christophgil]

Ok, thanks.

…

On Mon, Feb 6, 2023 at 9:59 AM JuniorJPDJ ***@***.***> wrote: Don't use my fork of ZipROFS - it is out of date and merged to branches here. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRZU6DPACSND2SLOMRBZ6LWWDDOVANCNFSM6AAAAAAUMLVTNU> . You are receiving this because you authored the thread.Message ID: ***@***.***>