XRootDPyFS

XRootDPyFS is a PyFilesystem-like interface to XRootD.

XRootD protocol aims at giving high performance, scalable fault tolerant access to data repositories of many kinds. The XRootDPyFS adds a high-level interface on top of the existing Python interface (pyxrootd) and makes it easy to e.g. copy a directory in parallel or recursively remove a directory.

Important

Since version 3.0.0, the dependency on PyFilesystem2 pyfs has been removed. The XRootDPyFS class is now a standalone implementation of the PyFilesystem2. The necessary classes and utilities have been added to this package. The only breaking change is the removal for the support of the Opener.

Further documentation is available on https://xrootdpyfs.readthedocs.io/.

Getting started

If you just want to try out the library, the easiest is to use Docker.

Build the image:

$ docker build --platform linux/amd64 -t xrootd .

Run the container and launch xrootd:

$ docker run --platform linux/amd64 -h xrootdpyfs -it xrootd bash

You will see the logs in the stdout. Next, in another shell, connect the container and fire up an ipython shell:

$ docker ps  # find the container id
$ docker exec -it <container-id> bash
[xrootdpyfs@xrootdpyfs code]$ ipython

Quick examples

Here is a quick example of a file listing with the xrootd PyFilesystem integration:

>>> from xrootdpyfs import XRootDPyFS
>>> fs = XRootDPyFS("root://localhost//tmp/")
>>> fs.listdir("xrootdpyfs")
['test.txt']

Reading files:

>>> f = fs.open("xrootdpyfs/test.txt")
>>> f.read()
b'Hello XRootD!\n'
>>> f.close()

Reading files using the readtext() method:

>>> fs.readtext("xrootdpyfs/test.txt")
b'Hello XRootD!\n'

Writing files:

>>> f = fs.open("xrootdpyfs/hello.txt", "w+")
>>> f.write("World")
>>> f.close()

Writing files using the writetext() method:

>>> fs.writetext("xrootdpyfs/test.txt", "World")

Development

The easiest way to develop is to build the Docker image and mount the source code as a volume to test any code modification with a running XRootD server:

$ docker build --platform linux/amd64 -t xrootd --progress=plain .
$ docker run --platform linux/amd64 -h xrootdpyfs -it -v <absolute path to this project>:/code xrootd bash
[xrootdpyfs@xrootdpyfs code]$ xrootd

In another shell:

$ docker ps  # find the container id
$ docker exec -it <container-id> bash
[xrootdpyfs@xrootdpyfs code]$ python -m pytest -vvv tests

If you want to test a specific version of xrootd, run:

$ docker build --platform linux/amd64 --build-arg xrootd_version=4.12.7 -t xrootd --progress=plain .

Documentation

Documentation is available at <http://xrootdpyfs.readthedocs.io/> or can be build using Sphinx:

pip install Sphinx
python setup.py build_sphinx

Testing

Running the tests are most easily done using docker:

$ docker build --platform linux/amd64 -t xrootd . && docker run --platform linux/amd64 -h xrootdpyfs -it xrootd