Fix async client timeout and cancellation issues

[anuraaga]

• Shield httpx response close from timeout so its closing of response iterators happens without interruption
• Separate tasks for executing httpx requests and retrieve responses through a queue
  • httpx does not seem to reliably cancel I/O, so to be able to return from the client quickly when cancelled, this seems like the only pattern to allow it. In some cases the I/O will still make it through to completion, but at least the client isn't blocked on it
• Improve timing of cancel_after_close to be as close to after close as possible. Before it was run right away
• Don't write TLS certs to files on event loop which would block it. I found it while seeing if it was just a busy event loop causing flakiness before the above findings
• Run client conformance tests in parallel to speed them up and do some stress testing of the client, which is what exposed the above bugs

[anuraaga]

Ok @connectrpc/python sorry for the churn. Initially all I wanted was to speed up client conformance tests, but that also ended up being a stress test of the client which we haven't really done before (it's only num cpus * 4, in the future we may need even more stressful tests) - this showed some possible bugs that could surface to users, albeit for corner cases. And while I was at it, we already had client cancellation tests marked as flaky, but I could finally deep dive into them and I think they're actually fixed, at least as best as we can do.

The change is large and complicated so it will be good to get eyes on it - @spenczar especially, I wonder if you've ran into patterns for dealing with async and cancellations cleanly before, maybe there's something besides the producer/consumer approach I took.

[anuraaga]

This is still flaky, I suspect it is an issue in the test harness, not code. I will defer debugging it as already did a lot of debugging and want to get the fixes in

[stefanvanburen]

stamping so we can get this in for #79; looks reasonable to me but not as familiar with the asyncio.Queue approach so up to you if you want to wait for a more thorough review.

httpx does not seem to reliably cancel I/O

Is there an upstream issue we can track? Or ought we open one (maybe hard to pinpoint)?

[anuraaga]

Thanks @stefanvanburen - it puts into perspective that this is a large workaround for likely an upstream issue. I am going to go ahead and defer this to another release to confirm it really is worth doing.