Synchronous version of `ngx.fetch`

[s3rj1k]

Additional synchronous version of ngx.fetch or any other curl-like HTTP request-response function that can be used in js_*_filter directive.

Function should be as close as possible to async ngx.fetch but work as a blocking sync operation.

[xeioex]

Hi @s3rj1k,

I doubt that we will implement synchronous version of ngx.fetch(). Instead please share your use case so we will think on how to use async ngx.fetch() there. You can start from here to get some ideas.

[s3rj1k]

Well the use-case is simple, change response headers based on external service response, it seems that the only place that this can be done is in js_header_filter but it does not support async ngx.fetch()/subrequest, so effectively no way of external communication is possible apart from fs.* and that is a messy solution in this specific case.

[xeioex]

@s3rj1k Please, elaborate more, what header and what information you send to an external service.

One possible workaround is to use r.subrequest() for the main request, modify the subrequest response asynchronously, send the modifies reply to the client. It works in some cases but it can be brittle in more complex configurations.

[s3rj1k]

I need to send response headers out to external service and that service will decide based on headers value if there is a need to modify them, so in this case I need that headers that would be sent out would be processed late in nginx filter chain (if I understand correctly nginx semantics).

So doing something with headers during js_content is a bit early as I understand.

[s3rj1k]

@xeioex Any news on that?

P.S. This will benefit Nginx Unit, as for it there is no alternative (harder) ways of doing similar.

[xeioex]

Hi @s3rj1k,

no progress on my side.

Also, ngx.fetch() is a part of nginx module (ie is tightly integrated with nginx and will not work without nginx). So it will not benefit Unit anyway.

[jo-carter]

I need to send response headers out to external service and that service will decide based on headers value if there is a need to modify them, so in this case I need that headers that would be sent out would be processed late in nginx filter chain (if I understand correctly nginx semantics).

So doing something with headers during js_content is a bit early as I understand.

@s3rj1k
I fail to see why you could not use js_content for this. You're able to perform the checks against external service using a response header header sourced from subrequest/fetch made for main request to backend, but before copying the header to the r.headersOut and exiting js_content handler function.

[s3rj1k]

I fail to see why you could not use js_content for this.

buffering content inside JS runtime would impact performance vs doing same in C inside filter phase.

[jo-carter]

@s3rj1k As I understand it, if you use subrequests for the main request to the backend(rather than fetch), then the njs module does not in fact buffer the response body. It's buffered by nginx (see subrequest_output_buffer_size).

It will of course consume more memory naturally than regular proxy_pass on it's own due it's ability to do partial buffering / buffer reuse as the client consumes data, however this rarely matters unless responses are expected to be very large.

Anyway I'd recommend benchmarking it to see if it fits your use-case if you have not done so, it's quite fast unless you start actually manipulating response body / perform complex logic in within js, from previous experience.