ASGI zipfly on Django 4.2 first packages the file(s) to memory, then sends to client

[T-101]

I've tried multiple times to make this work but I've ran out of options. So this is more like a cry for help.

Kinda sucks, since the same application this happens on, would benefit from having websockets enabled.

[sandes]

Without seeing the code, I really couldn't tell you what the problem is.
Having said that, the easy solution is to separate the downloads from the app itself.

[T-101]

The same code works on both WSGI and ASGI, and it produces the zipfile as expected. The only difference is that in ASGI mode it first creates the zipfile in memory and then sends it to client.

Tried fooling around with this today. It seems to boil down to Django's StreamingHttpResponse not accepting synchronous iterators when in ASGI mode. It throws this warning:

/usr/local/lib/python3.11/site-packages/django/http/response.py:534: Warning: StreamingHttpResponse must consume synchronous iterators in order to serve them asynchronously. Use an asynchronous iterator instead.

Spent a few hours trying to splash async/await and @sync_to_async decorators all around the code, but alas; no success yet.

[Va1a]

@T-101 did you ever have any luck with your ASGI fix? I was also planning on using Zipfly in an ASGI context, but obviously this bug would defeat the purpose. Or has a recent django version fixed it?

[T-101]

Nope, didn't get it working and I stopped trying ages ago. But since I had my Django running in a container, I was able to do a workaround. I added a Flask container and ran Zipfly there. Not optimal, but it got the job done and I was able to move on.

[adamsol]

If anyone is still looking for a solution - it's to create an async iterator from the synchronous one, like this (based on https://stackoverflow.com/a/76991813):

# sync_iterator = ZipFly(...).generator()

async def async_generator():
    while (chunk := await asyncio.to_thread(next, sync_iterator, None)) is not None:
        yield chunk

Then async_generator() can be fed into StreamingHttpResponse in an async environment, and the content will be streamed instead of being stored in memory.

I found that it's also worth increasing the default ZipFly's chunksize (e.g. to 1 MB) for better performance when streaming large files.

However, with a lot of small files, I'm hitting a noticeable slowdown near the end of the stream. I think it's related to an issue that has already been fixed: #112. @sandes, is there a new version release on the horizon?