Update dependencies #499
Update dependencies #499
Conversation
fsspec was initially added without the [s3] extra due to its recent addition.¹ It's been around for nearly 5 years now, so it should be safe. In addition, the boto3 dependency can be resolved without issues by newer versions of pip. Our minimum pip version target is whatever is bundled with Python 3.8, checked via CI. ¹ "Declare dependency on s3fs since we use fsspec's S3 support" (d1e54c3)
Version 1 was implicitly resolved with the s3fs[boto3] pin removed in the previous commit. It must be explicitly pinned since version 2 affects usage in two ways: 1. It renames ObjectProxy to BaseObjectProxy. 2. It adds type stubs, but only for Python >=3.10. Since Pyright is configured to target 3.8, it would error with v2 upon finding a stubs file without a type definition for the import. https://wrapt.readthedocs.io/en/latest/changes.html#version-2-0-0
Version 2 renames ObjectProxy to BaseObjectProxy. It also adds a type stubs file, but only defines types for Python >=3.10. Since Pyright is configured to target 3.8, a comment is added to ignore the error caused by lack of type definitions. https://wrapt.readthedocs.io/en/latest/changes.html#version-2-0-0
victorlin
force-pushed
the
victorlin/update-dependencies
branch
from
b0e88ba to
a4fea46
Compare
| # error described in https://github.com/fsspec/s3fs/issues/790 | ||
| "fsspec !=2023.9.1", | ||
| "s3fs[boto3] >=2021.04.0, !=2023.9.1, <2025.12.0", | ||
| "fsspec[s3] !=2023.9.1", |
There was a problem hiding this comment.
I tested some aws-batch commands with the CI build and seeing a warning that s3fs is very old
$ curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/mac | bash -s ci-build/20610155097
$ nextstrain build --aws-batch --attach c79d54ab-ae3c-4ca7-9e68-d2d90b3e1c35 --download logs/** --no-logs .
Attaching to Nextstrain AWS Batch Job ID: c79d54ab-ae3c-4ca7-9e68-d2d90b3e1c35
Job is SUCCEEDED
Job SUCCEEDED after 395.4 minutes (exited 0)
Downloading select files modified by job to /Users/joverlee/Repos/nextstrain/tb
- logs/**
fsspec.registry:301: UserWarning: Your installed version of s3fs is very old and known to cause
severe performance issues, see also https://github.com/dask/dask/issues/10276
To fix, you should specify a lower version bound on s3fs, or
update the current installation.Looking at the CI logs, this is installing s3fs-0.4.2. I wonder if declaring the s3fs depedency separately would help?
There was a problem hiding this comment.
Dang, good catch. The fsspec[s3] extra doesn't impose any version constraints on s3fs. I wonder why it's resolving to an older version instead of latest. I'll restore the s3fs minimum version constraint to see if that fixes things.
There was a problem hiding this comment.
I wondered if Python 3.8 was causing the downgrade, but even the tests with Python 3.13 are installing older versions of s3fs...
Looks like aiobotocore relaxed botocore + boto3 dependencies starting with 3.0.0, but latest release of s3fs (2025.12.0) still pins aiobotocore>=2.5.4,<3.0.0 (source). Maybe we just need to wait on the new release of s3fs?
There was a problem hiding this comment.
Yes, I hope a new version of s3fs that supports aiobotocore 3.x will help. The current pin is incompatible with versions of boto3 and wrapt resolved initially based on our project's dependency list. This first round of version resolutions is prioritized by pip, so it backtracks on s3fs until it finds version 0.4.2 which pip thinks can be installed with aiobotocore 3.x since it predates any aiobotocore version restriction.
Reverting a4fea46 and tightly pinning boto3 ~=1.41.0 fixes things, but this is the exact problem that's being described in the lengthy comment that we're trying to remove.
Description of proposed changes
This PR improves dependency declarations for fsspec, boto3, and wrapt. See commits for details.
Related issue(s)
Closes #496
Checklist