Improved docstrings and docs for Blob

Author: rwb27

docs/source/blobs.rst

@@ -122,14 +122,30 @@ Using :class:`.Blob` objects as inputs
 
 Because :class:`.Blob` outputs are represented in JSON as links, they are downloaded with a separate HTTP request if needed. There is currently no way to create a :class:`.Blob` on the server via HTTP, which means remote clients can use :class:`.Blob` objects provided in the output of actions but they cannot yet upload data to be used as input. However, it is possible to pass the URL of a :class:`.Blob` that already exists on the server as input to a subsequent Action. This means, in the example above of raw image capture, a remote client over HTTP can pass the raw :class:`.Blob` to the conversion action, and the raw data need never be sent over the network.
 
+
+HTTP interface and serialization
+--------------------------------
+
+:class:`.Blob` objects are subclasses of `pydantic.BaseModel`, which means they can be serialized to JSON and deserialized from JSON. When this happens, the :class:`.Blob` is represented as a JSON object with :prop:`.Blob.url` and :prop:`.Blob.content_type` fields. The :prop:`.Blob.url` field is a link to the data. The :prop:`.Blob.content_type` field is a string representing the MIME type of the data. It is worth noting that models may be nested: this means an action may return many :class:`.Blob` objects in its output, either as a list or as fields in a :class:`pydantic.BaseModel` subclass. Each :class:`.Blob` in the output will be serialized to JSON with its URL and content type, and the client can then download the data from the URL, one download per :class:`.Blob` object.
+
+When a :class:`.Blob` is serialized, the URL is generated with a unique ID to allow it to be downloaded. The URL is not guaranteed to be permanent, and should not be used as a long-term reference to the data. The URL will expire after 5 minutes, and the data will no longer be available for download after that time.
+
+In order to run an action and download the data, currently an HTTP client must:
+
+* Call the action that returns a :class:`.Blob` object, which will return a JSON object representing the invocation.
+* Poll the invocation until it is complete, and the :class:`.Blob` is available in its ``output`` property with the URL and content type.
+* Download the data from the URL in the :class:`.Blob` object, which will return the binary data.
+
+It may be possible to have actions return binary data directly in the future, but this is not yet implemented.
+
+
 Memory management and retention
 -------------------------------
 
 Management of :class:`.Blob` objects is currently very basic: when a :class:`.Blob` object is returned in the output of an Action that has been called via the HTTP interface, a fixed 5 minute expiry is used. This should be improved in the future to avoid memory management issues. 
 
+When a :class:`.Blob` is serialized, a URL is generated with a unique ID to allow it to be downloaded. However, only a weak reference is held to the :class:`.Blob`. Once an Action has finished running, the only strong reference to the :class:`.Blob` should be held by the output property of the action invocation. The :class:`.Blob` should be garbage collected once the output is no longer required, i.e. when the invocation is discarded - currently 5 minutes after the action completes, once the maximum number of invocations has been reached or when it is explicitly deleted by the client.
+
 The behaviour is different when actions are called from other actions. If `action_a` calls `action_b`, and `action_b` returns a :class:`.Blob`, that :class:`.Blob` will be subject to Python's usual garbage collection rules when `action_a` ends - i.e. it will not be retained unless it is included in the output of `action_a`.
 
-HTTP interface and serialization
---------------------------------
 
-:class:`.Blob` objects are subclasses of `pydantic.BaseModel`, which means they can be serialized to JSON and deserialized from JSON. When this happens, the :class:`.Blob` is represented as a JSON object with two fields: `url` and `content_type`. The `url` field is a link to the data. The `content_type` field is a string representing the MIME type of the data. When a :class:`.Blob` is serialized, a URL is generated with a unique ID to allow it to be downloaded. However, only a weak reference is held to the :class:`.Blob`. Once an Action has finished running, the only strong reference to the :class:`.Blob` should be held by the output property of the action invocation. The :class:`.Blob` should be garbage collected once the output is no longer required, i.e. when the invocation is discarded - currently 5 minutes after the action completes, once the maximum number of invocations has been reached or when it is explicitly deleted by the client.

docs/source/conf.py

@@ -27,6 +27,7 @@
 
 autodoc2_packages = ["../../src/labthings_fastapi"]
 autodoc2_render_plugin = "myst"
+autodoc2_class_docstring = "both"
 
 # autoapi_dirs = ["../../src/labthings_fastapi"]
 # autoapi_ignore = []

docs/source/index.rst

@@ -12,7 +12,7 @@ Documentation for LabThings-FastAPI
    dependencies/dependencies.rst
    blobs.rst
    concurrency.rst
-   client_code.rst
+   using_things.rst
 
    apidocs/index
 

src/labthings_fastapi/outputs/blob.py

@@ -1,22 +1,45 @@
-"""BLOB Output Module
+"""
+# BLOB Output Module
 
 The BlobOutput class is used when you need to return something file-like that can't
 easily (or efficiently) be converted to JSON. This is useful for returning large objects
 like images, especially where an existing file-type is the obvious way to handle it.
 
+There is a [dedicated documentation page on blobs](/blobs.rst) that explains how to use
+this mechanism.
+
 To return a file from an action, you should declare its return type as a BlobOutput
-subclass, defining the `media_type` attribute.
+subclass, defining the
+[`media_type`](#labthings_fastapi.outputs.blob.Blob.media_type) attribute.
+
+```python
+class MyImageBlob(Blob):
+    media_type = "image/png"
+
+class MyThing(Thing):
+    @thing_action
+    def get_image(self) -> MyImageBlob:
+        # Do something to get the image data
+        data = self._get_image_data()
+        return MyImageBlob.from_bytes(data)
+```
 
 The action should then return an instance of that subclass, with data supplied
 either as a `bytes` object or a file on disk. If files are used, it's your
-responsibility to ensure the file is deleted after the `BlobOutput` object is
-garbage-collected. Constructing it using the class methods `from_bytes` or
-`from_temporary_directory` will ensure this is done for you.
+responsibility to ensure the file is deleted after the
+[`Blob`](#labthings_fastapi.outputs.blob.Blob) object is
+garbage-collected. Constructing it using the class methods
+[`from_bytes`](#labthings_fastapi.outputs.blob.Blob.from_bytes) or
+[`from_temporary_directory`](#labthings_fastapi.outputs.blob.Blob.from_temporary_directory)
+will ensure this is done for you.
 
 Bear in mind a `tempfile` object only holds a file descriptor and is not safe for
-concurrent use: action outputs may be retrieved multiple times after the action has
-completed. Creating a temp folder and making a file inside it is the safest way to
-deal with this.
+concurrent use, which does not work well with the HTTP API:
+action outputs may be retrieved multiple times after the action has
+completed, possibly concurrently. Creating a temp folder and making a file inside it
+with
+[`from_temporary_directory`](#labthings_fastapi.outputs.blob.Blob.from_temporary_directory)
+is the safest way to deal with this.
 """
 
 from __future__ import annotations
@@ -53,10 +76,17 @@
 @runtime_checkable
 class BlobData(Protocol):
     """The interface for the data store of a Blob.
-    
-    :class:`.Blob` objects can represent their data in various ways. Each of
+
+    [`Blob`](#labthings_fastapi.outputs.blob.Blob) objects can represent their data in various ways. Each of
     those options must provide three ways to access the data, which are the
     `content` property, the `save()` method, and the `open()` method.
+
+    This protocol defines the interface needed by any data store used by a
+    [`Blob`](#labthings_fastapi.outputs.blob.Blob).
+
+    Objects that are used on the server will additionally need to implement the
+    [`ServerSideBlobData`](#labthings_fastapi.outputs.blob.ServerSideBlobData) protocol,
+    which adds a `response()` method and `id` property.
     """
 
     @property
@@ -80,10 +110,14 @@ def open(self) -> io.IOBase:
 
 class ServerSideBlobData(BlobData, Protocol):
     """A BlobData protocol for server-side use, i.e. including `response()`
-    
-    :class:`Blob` objects returned by actions must use :class:`.BlobData` objects
-    that can be downloaded. This protocol extends the :class:`.BlobData` protocol to
-    include a :meth:`~.ServerSideBlobData.response()` method that returns a FastAPI response object.
+
+    [`Blob`](#labthings_fastapi.outputs.blob.Blob) objects returned by actions must use
+    [`BlobData`](#labthings_fastapi.outputs.blob.BlobData) objects
+    that can be downloaded. This protocol extends that protocol to
+    include a [`response()`](#labthings_fastapi.outputs.blob.ServerSideBlobData.response) method that returns a FastAPI response object.
+
+    See [`BlobBytes`](#labthings_fastapi.outputs.blob.BlobBytes) or
+    [`BlobFile`](#labthings_fastapi.outputs.blob.BlobFile) for concrete implementations.
     """
 
     id: Optional[uuid.UUID] = None
@@ -157,13 +191,12 @@ def response(self) -> Response:
 class Blob(BaseModel):
     """A container for binary data that may be retrieved over HTTP
 
-    See :doc:`blobs` for more information on how to use this class.
-    
-    A :class:`.Blob` may be created to hold data using the class methods 
+    See the [documentation on blobs](/blobs.rst) for more information on how to use this class.
+
+    A [`Blob`](#labthings_fastapi.outputs.blob.Blob) may be created
+    to hold data using the class methods
     `from_bytes` or `from_temporary_directory`. The constructor will
-    attempt to deserialise a Blob from a URL, and may only be used within
-    a `blob_serialisation_context_manager`. This is made available when
-    actions are invoked, or when their output is returned.
+    attempt to deserialise a Blob from a URL (see `__init__` method).
 
     You are strongly advised to subclass this class and specify the
     `media_type` attribute, as this will propagate to the auto-generated
@@ -182,21 +215,29 @@ class Blob(BaseModel):
     )
 
     _data: Optional[ServerSideBlobData] = None
-    """This object holds the data, either in memory or as a file."""
+    """This object holds the data, either in memory or as a file.
+    
+    If `_data` is `None`, then the Blob has not been deserialised yet, and the
+    `href` should point to a valid address where the data may be downloaded.
+    """
 
     @model_validator(mode="after")
     def retrieve_data(self):
         """Retrieve the data from the URL
-        
-        When a :class:`.Blob` is created using its constructor, :mod:`pydantic`
+
+        When a [`Blob`](#labthings_fastapi.outputs.blob.Blob) is created
+        using its constructor, [`pydantic`](https://docs.pydantic.dev/latest/)
         will attempt to deserialise it by retrieving the data from the URL
-        specified in `href`. Currently, this must be a URL pointing to a 
-        :class:`.Blob` that already exists on this server.
+        specified in `href`. Currently, this must be a URL pointing to a
+        [`Blob`](#labthings_fastapi.outputs.blob.Blob) that already exists on
+        this server.
 
         This validator will only work if the function to resolve URLs to
-        :class:`.BlobData` objects has been set in the context variable. This
-        is done when actions are being invoked over HTTP, or when
-        their outputs are being returned.
+        [`BlobData`](#labthings_fastapi.outputs.blob.BlobData) objects
+        has been set in the context variable
+        [`url_to_blobdata_ctx`](#labthings_fastapi.outputs.blob.url_to_blobdata_ctx).
+        This is done when actions are being invoked over HTTP by the
+        [`BlobIOContextDep`](#labthings_fastapi.outputs.blob.BlobIOContextDep) dependency.
         """
         if self.href == "blob://local":
             if self._data:
@@ -216,17 +257,19 @@ def retrieve_data(self):
     @model_serializer(mode="plain", when_used="always")
     def to_dict(self) -> Mapping[str, str]:
         """Serialise the Blob to a dictionary and make it downloadable
-        
-        When :mod:`pydantic` serialises this object, it will call this method
-        to convert it to a dictionary. There is a significant side-effect, which
-        is that we will add the blob to the :class:`.BlobDataManager` so it 
+
+        When [`pydantic`](https://docs.pydantic.dev/latest/) serialises this object,
+        it will call this method to convert it to a dictionary. There is a
+        significant side-effect, which is that we will add the blob to the
+        [`BlobDataManager`](#labthings_fastapi.outputs.blob.BlobDataManager) so it
         can be downloaded.
-        
-        This serialiser will only work if the function to resolve URLs to
-        :class:`.BlobData` objects has been set in the context variable. This
-        is done when the outputs of actions are being returned.
 
-        Note that the 
+        This serialiser will only work if the function to assign URLs to
+        [`BlobData`](#labthings_fastapi.outputs.blob.BlobData) objects
+        has been set in the context variable
+        [`blobdata_to_url_ctx`](#labthings_fastapi.outputs.blob.blobdata_to_url_ctx).
+        This is done when actions are being returned over HTTP by the
+        [`BlobIOContextDep`](#labthings_fastapi.outputs.blob.BlobIOContextDep) dependency.
         """
         if self.href == "blob://local":
             try:
@@ -249,10 +292,30 @@ def to_dict(self) -> Mapping[str, str]:
 
     @classmethod
     def default_media_type(cls) -> str:
+        """The default media type.
+
+        `Blob` should generally be subclassed to define the default media type,
+        as this forms part of the auto-generated documentation. Using the
+        `Blob` class directly will result in a media type of `*/*`, which makes
+        it unclear what format the output is in.
+        """
         return cls.model_fields["media_type"].get_default()
 
     @property
     def data(self) -> ServerSideBlobData:
+        """The data store for this Blob
+
+        `Blob` objects may hold their data in various ways, defined by the
+        [`ServerSideBlobData`](#labthings_fastapi.outputs.blob.ServerSideBlobData)
+        protocol. This property returns the data store for this `Blob`.
+
+        If the `Blob` has not yet been downloaded, there may be no data
+        held locally, in which case this function will raise a `ValueError`.
+
+        It is recommended to use the `content` property or `save()` or `open()`
+        methods rather than accessing this property directly. Those methods will
+        download data if required, rather than raising an error.
+        """
         if self._data is None:
             raise ValueError("This Blob has no data.")
         return self._data
@@ -329,7 +392,16 @@ def response(self):
 
 
 def blob_type(media_type: str) -> type[Blob]:
-    """Create a BlobOutput subclass for a given media type"""
+    """Create a BlobOutput subclass for a given media type
+
+    This convenience function may confuse static type checkers, so it is usually
+    clearer to make a subclass instead, e.g.:
+
+    ```python
+    class MyImageBlob(Blob):
+        media_type = "image/png"
+    ```
+    """
     if "'" in media_type or "\\" in media_type:
         raise ValueError("media_type must not contain single quotes or backslashes")
     return create_model(
@@ -342,17 +414,20 @@ def blob_type(media_type: str) -> type[Blob]:
 class BlobDataManager:
     """A class to manage BlobData objects
 
-    The BlobManager is responsible for serving `Blob` objects to clients. It
+    The `BlobManager` is responsible for serving `Blob` objects to clients. It
     holds weak references: it will not retain `Blob`s that are no longer in use.
-    Most `Blob`s will be retained"""
+    Most `Blob`s will be retained by the output of an action: this holds a strong
+    reference, and will be expired by the
+    [`ActionManager`](#labthings_fastapi.actions.ActionManager).
+    """
 
     _blobs: WeakValueDictionary[uuid.UUID, ServerSideBlobData]
 
     def __init__(self):
         self._blobs = WeakValueDictionary()
 
     def add_blob(self, blob: ServerSideBlobData) -> uuid.UUID:
-        """Add a BlobOutput to the manager"""
+        """Add a BlobOutput to the manager, generating a unique ID"""
         if hasattr(blob, "id") and blob.id is not None:
             if blob.id in self._blobs:
                 return blob.id
@@ -380,12 +455,29 @@ def attach_to_app(self, app: FastAPI):
 
 
 blobdata_to_url_ctx = ContextVar[Callable[[ServerSideBlobData], str]]("blobdata_to_url")
+"""This context variable gives access to a function that makes BlobData objects
+downloadable, by assigning a URL and adding them to the 
+[`BlobDataManager`](#labthings_fastapi.outputs.blob.BlobDataManager).
+
+It is only available within a 
+[`blob_serialisation_context_manager`](#labthings_fastapi.outputs.blob.blob_serialisation_context_manager)
+because it requires access to the `BlobDataManager` and the `url_for` function
+from the FastAPI app.
+"""
 
 url_to_blobdata_ctx = ContextVar[Callable[[str], BlobData]]("url_to_blobdata")
+"""This context variable gives access to a function that makes BlobData objects
+from a URL, by retrieving them from the
+[`BlobDataManager`](#labthings_fastapi.outputs.blob.BlobDataManager).
+
+It is only available within a 
+[`blob_serialisation_context_manager`](#labthings_fastapi.outputs.blob.blob_serialisation_context_manager)
+because it requires access to the `BlobDataManager`.
+"""
 
 
 async def blob_serialisation_context_manager(request: Request):
-    """Set context variables to allow blobs to be serialised"""
+    """Set context variables to allow blobs to be [de]serialised"""
     thing_server = find_thing_server(request.app)
     blob_manager: BlobDataManager = thing_server.blob_data_manager
     url_for = request.url_for
@@ -415,3 +507,4 @@ def url_to_blobdata(url: str) -> BlobData:
 BlobIOContextDep: TypeAlias = Annotated[
     BlobDataManager, Depends(blob_serialisation_context_manager)
 ]
+"""A dependency that enables `Blob`s to be serialised and deserialised."""