Add more conceptual documentation

Author: rwb27

docs/source/client_code.rst

@@ -0,0 +1,18 @@
+Client code
+===========
+
+The interface to a `Thing` is defined by its interaction affordances, which are defined in the Thing Description. The `labthings-fastapi.client` library provides a `ThingClient` class to interact with a `Thing` via HTTP. This is a class with a method for each Action, and a property for each Property of the Thing. The intention is to provide a simple, pythonic interface that plays nicely with IDEs and autocompletion. 
+
+An additional goal is to provide an interface that is consistent between the server and client code: a `DirectThingClient` class is used by the `labthings-fastapi` server to call actions and properties of other `Thing`s, which means code for an action may be developed as an HTTP client, for example in a Jupyter notebook, and then moved to the server with minimal changes. Currently, there are a few differences in behaviour between local and remote `Thing`s, most notably the return types (which are usually Pydantic models on the server, and currently dictionaries generated from JSON on the client). This should be improved in the future.
+
+Client code generation
+----------------------
+
+Currently, most clients are created using the class method `ThingClient.from_url`. This returns an instance of a dynamically-created subclass, rather than a `ThingClient` instance directly. The subclass is required in order to add methods and properties corresponding to the Thing Description sent by the server. While this is a solution that should work immediately, it does not work well with code completion or static analysis, and client objects must be introspected on-the-fly.
+
+In the future, `labthings_fastapi` will generate custom client subclasses. These will have the methods and properties defined in a Python module, including type annotations. This will allow static analysis (e.g. with MyPy) and IDE autocompletion to work. Most packages that provide a `Thing` subclass will want to release a client package that is generated automatically in this way. The intention is to make it possible to add custom Python code to this client, for example to handle specialised return types more gracefully or add convenience methods.
+
+
+
+
+

docs/source/concurrency.rst

@@ -0,0 +1,15 @@
+Concurrency in `labthings-fastapi`
+==================================
+
+One of the major challenges when controlling hardware, particularly from web frameworks, is concurrency. Most web frameworks assume resources (database connections, object storage, etc.) may be instantiated multiple times, and often initialise or destroy objects as required. In contrast, hardware can usually only be controlled from one process, and usually is initialised and shut down only once.
+
+`labthings-fastapi` instantiates each `Thing` only once, and runs all code in a thread. More specifically, each time an action is invoked via HTTP, a new thread is created to run the action. Similarly, each time a property is read or written, a new thread is created to run the property method. This means that `Thing` code should protect important variables or resources using locks from the `threading` module, and need not worry about writing asynchronous code.
+
+In the case of properties, the HTTP response is only returned once the `Thing` code is complete. Actions currently return a response immediately, and must be polled to determine when they have completed. This behaviour may change in the future, most likely with the introduction of a timeout to allow the client to choose between waiting for a response or polling.
+
+Many of the functions that handle HTTP requests are asynchronous, running in an `anyio` event loop. This enables many HTTP connections to be handled at once with good efficiency. The interface between async and threaded code is provided by a "Blocking Portal" created when the LabThings server is started. A FastAPI Dependency allows the blocking portal to be obtained: while it's very unlikely more than one LabThings server will exist in one Python instance, we avoid referring to the blocking portal globally in an effort to avoid concurrency issues.
+
+If threaded code needs to call code in the `anyio` event loop, the blocking portal dependency should be used. There are relatively few occasions when `Thing` code will need to consider this explicitly: more usually the blocking portal will be obtained by a LabThings function, for example the `MJPEGStream` class.
+
+When one `Thing` calls the actions or properties of another `Thing`, either directly or via a `DirectThingClient`, no new threads are spawned: the action or property is run in the same thread as the caller. This mirrors the behaviour of the `ThingClient`, which blocks until the action or property is complete.
+

docs/source/index.rst

@@ -10,36 +10,43 @@ Welcome to labthings-fastapi's documentation!
    :maxdepth: 2
    :caption: Contents:
 
-   core_concepts.rst
+   wot_core_concepts.rst
    quickstart/quickstart.rst
    dependencies/dependencies.rst
+   concurrency.rst
+   client_code.rst
 
    apidocs/index
 
 `labthings-fastapi` implements a Web of Things interface for laboratory hardware using Python. This is a ground-up rewrite of python-labthings_, replacing Flask 1 and Marshmallow with FastAPI and Pydantic. It is the underlying framework for v3 of the `OpenFlexure Microscope software <https://gitlab.com/openflexure/openflexure-microscope-server/>`_.
 
-Features include:
+`labthings-fastapi` aims to simplify the process of making laboratory instruments available via an HTTP API. Key features and design aims are below:
 
-* Alignment with the `W3C Web of Things <https://www.w3.org/WoT/>`_ standard (see :doc:`core_concepts`)
+* Functionality together in `Thing` subclasses, which represent units of hardware or software (see :doc:`wot_core_concepts`)
+* Methods and properties of `Thing` subclasses may be added to the HTTP API and Thing Description using decorators
+* Vocabulary and concepts are aligned with the `W3C Web of Things <https://www.w3.org/WoT/>`_ standard (see :doc:`wot_core_concepts`)
     - Things are classes, with properties and actions defined exactly once
-    - Various improvements to TD generation and validation with `pydantic`
-* Cleaner API
+    - Thing Descriptions are automatically generated, and validated with `pydantic`
+    - OpenAPI documentation is automatically generated by FastAPI
+* We follow FastAPI_'s lead and try to use standard Python features to minimise unnecessary code
     - Datatypes of action input/outputs and properties are defined with Python type hints
     - Actions are defined exactly once, as a method of a `Thing` class
     - Properties and actions are declared using decorators (or descriptors if that's preferred)
-    - Dependency injection is used to manage relationships between Things and dependency on the server
-* Async HTTP handling
-    - Starlette (used by FastAPI) can handle requests asynchronously - potential for websockets/events (not used much yet)
-    - `Thing` code is still, for now, threaded. I intend to make it possible to write async things in the future, but don't intend it to become mandatory
-* Smaller codebase
-    - FastAPI more or less completely eliminates OpenAPI generation code from our codebase
-    - Thing Description generation is very much simplified by the new structure (multiple Things instead of one massive Thing with many extensions)
+    - FastAPI_ "Dependency injection" is used to manage relationships between Things and dependency on the server
+* Lifecycle and concurrency are appropriate for hardware: `Thing` code is always run in a thread, and each `Thing` is instantiated and shut down only once.
+    - Starlette (used by FastAPI) can handle requests asynchronously - this improves performance and enables websockets and other long-lived connections.
+    - `Thing` code is still, for now, threaded. In the future it may become possible to us other concurrency models in `Thing` code.
+
+Compared to `python-labthings`_, this framework updates dependencies, shrinks the codebase, and simplifies the API.
+* FastAPI more or less completely eliminates OpenAPI generation code from our codebase
+* Marshmallow schemas and endpoint classes are replaced with Python type hints, eliminating double- or triple-definition of actions and their inputs/outputs.
+* Thing Description generation is very much simplified by the new structure (multiple Things instead of one massive Thing with many extensions)
 
 
 Installation
 ------------
 
-``pip install labthings-fastapi``
+``pip install labthings-fastapi[server]``
 
 Indices and tables
 ==================
@@ -48,4 +55,6 @@ Indices and tables
 * :ref:`modindex`
 * :ref:`search`
 
-.. _python-labthings: https://github.com/labthings/python-labthings/
+.. _python-labthings: https://github.com/labthings/python-labthings/
+.. _FastAPI: https://fastapi.tiangolo.com/
+.. _pydantic: https://pydantic-docs.helpmanual.io/

docs/source/core_concepts.rst docs/source/wot_core_concepts.rstdocs/source/core_concepts.rst renamed to docs/source/wot_core_concepts.rst

@@ -1,7 +1,7 @@
-Core Concepts
-=============
+Web of Things Core Concepts
+===========================
 
-LabThings is rooted in the `W3C Web of Things standards <WoT>`_. Using IP networking in labs is not itself new, though perhaps under-used. However lack of proper standardisation has stiffled widespread adoption. LabThings, rather than try to introduce new competing standards, uses the architecture and terminology introduced by the W3C Web of Things. A full description of the core architecture can be found in the `Web of Things (WoT) Architecture <https://www.w3.org/TR/wot-architecture/#sec-wot-architecture>`_ document. However, a brief outline of the concepts relevant to `labthings-fastapi` is given below.
+LabThings is rooted in the `W3C Web of Things standards <WoT>`_. Using IP networking in labs is not new, though perhaps under-used. However lack of proper standardisation has stiffled widespread adoption. LabThings, rather than try to introduce new competing standards, uses the architecture and terminology introduced by the W3C Web of Things. A full description of the core architecture can be found in the `Web of Things (WoT) Architecture <https://www.w3.org/TR/wot-architecture/#sec-wot-architecture>`_ document. However, a brief outline of the concepts relevant to `labthings-fastapi` is given below.
 
 Thing
 ---------