Connection broken: IncompleteRead for dandi-cli; add a generic support for skipping some logs?

[yarikoptic]

Exclusively for dandi-cli started to happen recently starting with

  16     Jun 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-cli && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-cli.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch

logs from execution with traceback etc

Subject: Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-cli && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-cli.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch

2025-07-06T08:00:01-0400 [INFO    ] tinuous tinuous 0.6.0
2025-07-06T08:00:02-0400 [INFO    ] tinuous Fetching resources from github
2025-07-06T08:00:02-0400 [INFO    ] tinuous Fetching runs newer than 2025-07-05 06:04:35+00:00
2025-07-06T08:00:02-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/claude.yml (Claude Code)
2025-07-06T08:00:03-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/codeql.yml (CodeQL)
2025-07-06T08:00:03-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/docs.yml (Build Docs)
2025-07-06T08:00:04-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/labels.yml (Check PR Labels)
2025-07-06T08:00:05-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/lint.yml (Linters)
2025-07-06T08:00:05-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/release.yml (Auto-release on PR merge)
2025-07-06T08:00:06-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/run-tests.yml (Tests)
2025-07-06T08:00:06-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/test.yml (Tests)
2025-07-06T08:00:07-0400 [INFO    ] tinuous Found run 6563
2025-07-06T08:05:25-0400 [INFO    ] tinuous Downloading logs for test.yml (Tests) #6563 to 2025/07/06/github/cron/20250706T060452/03b7c07/Tests/6563
2025-07-06T08:05:55-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(3536 bytes read, 560 more expected)', IncompleteRead(3536 bytes read, 560 more expected)); waiting & retrying
2025-07-06T08:06:26-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4016 bytes read, 80 more expected)', IncompleteRead(4016 bytes read, 80 more expected)); waiting & retrying
2025-07-06T08:06:59-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4056 bytes read, 40 more expected)', IncompleteRead(4056 bytes read, 40 more expected)); waiting & retrying
2025-07-06T08:07:32-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(3856 bytes read, 240 more expected)', IncompleteRead(3856 bytes read, 240 more expected)); waiting & retrying
2025-07-06T08:08:07-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4008 bytes read, 88 more expected)', IncompleteRead(4008 bytes read, 88 more expected)); waiting & retrying
2025-07-06T08:08:43-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4016 bytes read, 80 more expected)', IncompleteRead(4016 bytes read, 80 more expected)); waiting & retrying
2025-07-06T08:09:19-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4048 bytes read, 48 more expected)', IncompleteRead(4048 bytes read, 48 more expected)); waiting & retrying
2025-07-06T08:09:56-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4080 bytes read, 16 more expected)', IncompleteRead(4080 bytes read, 16 more expected)); waiting & retrying
2025-07-06T08:10:35-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(8160 bytes read, 32 more expected)', IncompleteRead(8160 bytes read, 32 more expected)); waiting & retrying
2025-07-06T08:11:14-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4080 bytes read, 16 more expected)', IncompleteRead(4080 bytes read, 16 more expected)); waiting & retrying
2025-07-06T08:11:55-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(4008 bytes read, 88 more expected)', IncompleteRead(4008 bytes read, 88 more expected)); waiting & retrying
2025-07-06T08:12:36-0400 [WARNING ] tinuous Download of https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs interrupted: ('Connection broken: IncompleteRead(3760 bytes read, 336 more expected)', IncompleteRead(3760 bytes read, 336 more expected)); waiting & retrying
2025-07-06T08:13:18-0400 [ERROR   ] tinuous Max retries exceeded
Traceback (most recent call last):
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 441, in _error_catcher
    yield
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 770, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 723, in _handle_chunk
    returned_chunk = self._fp._safe_read(self.chunk_left)
  File "/home/dandi/miniconda3/lib/python3.8/http/client.py", line 610, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(4064 bytes read, 32 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/requests/models.py", line 760, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 575, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 796, in read_chunked
    self._original_response.close()
  File "/home/dandi/miniconda3/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/urllib3/response.py", line 458, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(4064 bytes read, 32 more expected)', IncompleteRead(4064 bytes read, 32 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/backup/dandi/tinuous-logs/venv/bin/tinuous", line 10, in <module>
    sys.exit(main())
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/__main__.py", line 127, in fetch
    paths = obj.download(Path(path))
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/github.py", line 356, in download
    self.client.download_zipfile(self.logs_url, path)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 143, in download_zipfile
    self.download(path, zippath)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 117, in download
    for chunk in r.iter_content(chunk_size=8192):
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/requests/models.py", line 763, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(4064 bytes read, 32 more expected)', IncompleteRead(4064 bytes read, 32 more expected))

using straight `wget` on my laptop we pretty much see similar problem which eventually leads to 403 Forbidden

$ wget -S --header="Authorization: Bearer XXXX" https://api.github.com/repos/dandi/dandi-cli/actions/runs/16095976609/logs
...
  X-RateLimit-Limit: 5000
  X-RateLimit-Remaining: 4932
  X-RateLimit-Reset: 1751875203
  X-RateLimit-Used: 68
  X-RateLimit-Resource: core
...
Resolving results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)... 140.82.114.22
Connecting to results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)|140.82.114.22|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  content-disposition: attachment; filename=logs_41239562373.zip
  Content-Type: application/zip
  Date: Mon, 07 Jul 2025 07:27:11 GMT
  Transfer-Encoding: chunked
  x-github-backend: Kubernetes
  X-GitHub-Request-Id: XXXX
Length: unspecified [application/zip]
Saving to: ‘logs’

logs                            [           <=>                            ]  78.20M  2.96MB/s    in 26s     

2025-07-07 03:27:41 (3.06 MB/s) - Read error at byte 82018304 (The request is invalid.).Retrying.

--2025-07-07 03:27:42--  (try: 2)  https://results-receiver.actions.githubusercontent.com/rest/runs/e0184fe9-1488-40fe-95b7-2a402630b958/logs?filename=logs_41239562373.zip&signature=XXX
Connecting to results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)|140.82.114.22|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  content-disposition: attachment; filename=logs_41239562373.zip
  Content-Type: application/zip
  Date: Mon, 07 Jul 2025 07:27:42 GMT
  Transfer-Encoding: chunked
  x-github-backend: Kubernetes
  X-GitHub-Request-Id: XXXX
Length: unspecified [application/zip]
Saving to: ‘logs’

logs                            [                                   <=>    ]  99.35M  3.89MB/s    in 30s     

2025-07-07 03:28:12 (3.32 MB/s) - Read error at byte 104173568 (The request is invalid.).Retrying.

--2025-07-07 03:28:14--  (try: 3)  https://results-receiver.actions.githubusercontent.com/rest/runs/e0184fe9-1488-40fe-95b7-2a402630b958/logs?filename=logs_41239562373.zip&signature=XXXX
Connecting to results-receiver.actions.githubusercontent.com (results-receiver.actions.githubusercontent.com)|140.82.114.22|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 403 Forbidden
  Date: Mon, 07 Jul 2025 07:28:15 GMT
  Content-Length: 0
  x-github-backend: Kubernetes
  X-GitHub-Request-Id: XXXX
2025-07-07 03:28:15 ERROR 403: Forbidden.

so I think we are just doomed to skip such runs/logs?

may be for now, to be explicit, we should just add a feature to explicitly skip them in the config, and make it flexible, e.g. some kind of a code eval based on f-string e.g.

ci:
  github:
    paths:
      logs: '{year}//{month}//{day}/{ci}/{type}/{type_id}/{commit[:7]}/{wf_name}/{number}/'
    skip_evals:
    - "f'{wf_name}/{number}' == 'Tests/6563'"