Stability of the API

[jborean93]

Hi, I'm wondering what the stability of the API is for httpcore. In particular I'm to initialize my own *HTTPConnection class in _async and _sync but I'm not sure whether the location of these classes, or even the structure, is expected to change in any future release. Can I safely use these classes with maybe a cap on httpcore<1.0.0 in my requirements or would you not recommend this?

I've got a very unique case where I need to tweak a transport for httpx and the *ConnectionPool classes just don't fit in my the way I need to connect, authenticate, and encrypt data for an endpoint. By being able to specify my own transport I can specify this behaviour myself but by using a "private" class like AsyncHTTPConnection I might be relying on unstable/private APIs that could change in a future release.

[florimondmanca]

I think you're referring to the discussion we had here? :) encode/httpx#1473

To be clear, right now the only thing that's public in HTTPCore is what is documented on the docs site. The internals of the connection pool or proxy implementations are private, and subject to change. Doesn't mean you can tweak them, but just be sure to pin HTTPCore strictly and expect to have to update them slightly as we fix bugs or tweak things internally.

I don't think we're ready to make those internals public in full, however I'm personally happy to discuss specific issues and things we could expose selectively to make it easier to customize connections.

[jborean93]

Yea, unfortunately I keep on hitting roadblocks as I progress further :(. Ultimately the authentication process and subsequent messages are tied directly with the connection which just isn't publicly exposed in httpx/httpcore. As you said at this point I'm effectively pinning the httpcore version to one where I know works. What I'm ultimately hoping to achieve is to expose the functionality I need as a public interface so I don't have to be so selective with my pinning at some point in the future.

From what I've worked with I've found that making a custom transport solves a lot of my existing issues but there remains 2 problems:

• Not being able to detect when a connection is dropped from a keep-alive expiry or server side disconnect
  • AsyncConnectionPool handles all the pool logic internally as well as building AsyncHTTPConnection through the private imports but the keep-alive/disconnect issue still remains
  • I can do the same thing myself but it means I need to import AsyncHTTPConnection through the private imports hence this issue
  • There's AsyncConnectionPool._create_connection() but that's also a private function so has the same issue with creating the connection object myself
• For the authentication protocols I need to use I need to include TLS connection specific data when generating the auth token
  • Currently the socket and TLS handshake is automatically done on the first AsyncHTTPConnection.arequest()
  • This is unfortunately too late as the TLS connection is closed on the first failed request, the next request with the auth token will be on a new connection/TLS context so the data may differ
  • The only real way to solve this is to be able to create the socket and do the TLS handshake before we send the first HTTP request
  • See Expose socket connection interface in connection #273

I could potentially copy AsyncHTTPConnection and use it internally but there's a lot of things that httpcore just handles that I'm hoping not to duplicate code.

[florimondmanca]

@jborean93 From the specificity of your use case, and especially what seems to be "I need a long-lived special TLS connection that shouldn't close upon errors", are you sure you need our connection pooling behavior? Were you able to build a prototype that solely uses an AsyncHTTPConnection as a transport (or perhaps a wrapper around a custimized version of it)?

The code for our HTTP/1.1 connection implementation isn't that big. It might be a very valid thing to do to duplicate it and tweak it to your liking. It may take some time to study and understand how it works specifically (reviewing the h11 docs helps), but it's not intractable. :-) And you'd be able to apply tweaks as you see fit, and remove bits you don't need.

Edit: actually, since AsyncHTTP11Connection takes a socket as input, you might be able to reuse it, and instead wrap it in a transport that does the "open socket" behavior you need. Note that you'd need to match it up to return an AsyncSocket, which is a private interface. If you duplicate AsyncHTTP11Connection to use your own socket type, then you're completely free wrt HTTPCore evolutions, and don't need any pinning. Or you can use HTTPCore's concurrency backends, but you'd need to pin. You can get a sense of what this might look like in this example back from when we were working on UDS support. :-)

[jborean93]

Were you able to build a prototype that solely uses an AsyncHTTPConnection as a transport (or perhaps a wrapper around a custimized version of it)?

I was with https://github.com/jborean93/pypsrp/blob/dfae26ed0fb6f6a27f8f7bc8ed121120bee75b3c/psrp/io/wsman.py#L40. Specially I create the connection and connect to the socket/TLS handshake with https://github.com/jborean93/pypsrp/blob/dfae26ed0fb6f6a27f8f7bc8ed121120bee75b3c/psrp/io/wsman.py#L211-L224. From there I check to see if it's still alive after a keep-alive expiry or whether the socket is readable (closed by server) and reconnect/reauth if needed.

It might be a very valid thing to do to duplicate it and tweak it to your liking.

That is true, I'm just hoping to avoid duplication of work as much as I can but it's probably the only sane thing that I can do without pinning to a specific version. Although as you said I need to copy more than just AsynHTTP11Connection, I also need to copy the backends, or at least parts of it to implement some of the functionality AsyncHTTP11Connection relies on. But it's probably for the best that I just copy and work in the background to expose what I need publicly for a future version.

Heres to hoping I can still use proxies/socks in some way when using my own socket, sounds like that will require even more logic :(