DecodingError: BrotliDecoderDecompressStream

[dalf]

I don't know if this is a bug: httpx can't decompress the response from brave.com when it is compressed with Brotli.

$ pip install httpx[cli,brotli]
$ httpx "https://search.brave.com/search?q=time&offset=2&spellcheck=1"
...
DecodingError: BrotliDecoderDecompressStream failed while processing the stream

Firefox or CURL don't have this issue:

$ curl -vvv --compressed "https://search.brave.com/search?q=time&offset=2&spellcheck=1"
...
< content-encoding: br
...

[lovelydinosaur]

Do either of requests or aiohttp reproduce the issue?

[dalf]

I can reproduce the issue with aiohttp, requests and pycurl on Ubuntu 20.04, brotli 1.0.7, so this is not a httpx / httpcore issue.

Source code:

httpx

import httpx

response = httpx.get('https://search.brave.com/search?q=time&offset=2&spellcheck=1', headers={'Accept-Encoding': 'gzip, deflate, br'})
print(response.text[:50])
print(response.headers['content-encoding'])

requests

# pip install requests brotli
import requests

response = requests.get('https://search.brave.com/search?q=time&offset=2&spellcheck=1', headers={'Accept-Encoding': 'br'})
print(response)

aiohttp

# pip install aiohttp[speedup]
import aiohttp
import asyncio

async def main():

    async with aiohttp.ClientSession(headers={'Accept-Encoding': 'br'}) as session:
        async with session.get('https://search.brave.com/search?q=time&offset=2&spellcheck=1') as response:
            print("Content-Encoding:", response.headers['Content-Encoding'])

            html = await response.text()
            print("Body:", html[:50], "...")

loop = asyncio.new_event_loop()
loop.run_until_complete(main())

pycurl

# pip install pycurl certifi brotli

from io import BytesIO

import pycurl
import certifi
import brotli

headers = {}

def header_function(header_line):
    if b':' in header_line:
        name, value = header_line.decode('iso-8859-1').split(':', 1)
        headers[name.lower()] = value[:-2].strip()

buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, 'https://search.brave.com/search?q=time&offset=2&spellcheck=1')
c.setopt(pycurl.HTTPHEADER, ['Accept-Encoding: gzip, deflate, br'])
c.setopt(c.WRITEDATA, buffer)
c.setopt(c.HEADERFUNCTION, header_function)
c.setopt(c.CAINFO, certifi.where())
c.perform()
c.close()

body = buffer.getvalue()
# Body is a byte string.
# We have to know the encoding in order to print it to a text file
# such as standard output.
# print(body)
print(headers.get('content-encoding'))
print(brotli.decompress(body))

There is the same issue on Alpine version 3.11 (Brotli 1.0.7-r3) and 3.16 (Brotli 1.0.9-r6):

Dockerfile

FROM alpine:3.11

RUN apk upgrade --no-cache \
 && apk add --no-cache -t build-dependencies \
    build-base \
    py3-setuptools \
    python3-dev \
    python3 \
    py3-pip \
    openssl \
    libffi-dev \
    openssl-dev \
    tar \
    brotli \
    git
RUN pip3 install httpx brotli

WORKDIR /tmp

COPY thttpx.py .

CMD python3 /tmp/thttpx.py

docker build -t thttpx .
docker run --rm -ti thttpx

There is no issue if I change the URL to https://search.brave.com

[lovelydinosaur]

Okay, thanks for that. I'm not going to dig into this myself, but looks like a good starting point for anyone who wants to try to figure out if this is a bug in the python brotli implementations that each of these tools are using, or if it's a bug in Brave's compression.