Which zstd package to use for Python < 3.14?

[stf976]

In src/litdata/raw/indexer.py the zstd module is used to compress index files. This module is expected to have an exception defined named ZstdError.

For Python 3.14 and above, the module is included with Python (https://docs.python.org/3/library/compression.zstd.html) and it includes this exception.

For earlier versions, the module is imported as import zstd. However the pip package zstd does not include ZstdError, only zstd.Error.

The pip package on which the Python implementation is based also contains ZstdError, but it would be imported as import pystd or import backports.zstd.

Am I missing something or is this a bug in litdata?

[anntnzrb]

This is a bug. The code in /src/litdata/raw/indexer.py catches zstd.ZstdError, but the zstd PyPI package only exposes zstd.Error, not ZstdError.

Fix: Replace zstd with pyzstd in requirements/test.txt:

pyzstd; python_version < "3.14"

Then update imports to import pyzstd as zstd for Python < 3.14. The pyzstd package provides ZstdError and has an API compatible with Python 3.14's compression.zstd module.

[bhimrazy]

Thanks @stf976 and @anntnzrb for pointing this out! 🙌

You're right — the fallback zstd package doesn’t expose ZstdError, so the current handling in indexer.py is inconsistent for Python < 3.14. I've opened an issue to track this: #766

We'll get a fix in soon. Thanks again for raising it!

[stf976]

Thank you @anntnzrb and @bhimrazy for the clarifications and fix!