add some design considerations

Author: robgee86

design/declarative.md

@@ -0,0 +1,40 @@
+# Declarative style design
+
+## Requirements
+
+1. Declarative programming style: the user declaratively defines the pipeline structure, e.g. via chaining or composition of the step classes.
+2. Java Stream API-like: handles parallelism, backpressure, and synchronization automatically.
+3. Invisible framework: the user code should not depend directly on the framework, if possible. The framework should wrap the logic and add asynchronicity.
+4. Maximum Freedom, Simplicity, Reusability: as a consequence, nodes should be easy to develop and usable outside the framework.
+5. Encapsulation: the user defines "nodes of computation" that are plain Python classes unaware of the framework, of their asynchronous execution and of other nodes.
+6. Single entry point: we want a single entry point, responsible for linking the nodes together, orchestration and handling parallelism, backpressure, and synchronization.
+7. Simple API: the API should be easy to understand and implement.
+
+## Design
+
+- Pipeline: the Pipeline class will take a sequence of source -> step -> sink classes as input. The order of these classes will define the order in the processing pipeline.
+- Source, Step & Sink: since the framework shouldn't impose any inheritance or interface to implement, we need a convention for how the framework will interact with these classes. A simple approach is to expect each node to have a specific method that the framework can call to process an item. Let's call this method `produce` for sources, `process` for steps and `consume` for sinks. This method should take one input and return one output (or an exception).
+- Data Flow: the framework will be responsible for passing data between the steps and should provide a way to capture the source data, process it in steps and consume the final output of the pipeline in the sink. This could involve using queues for internal communication and managing backpressure and asynchronicity. We also need to expose some sort of tuning of the backpressure strategy and adapters to manage the data model mismatch between these components.
+- Boot sequence: the Pipeline class will need a method to start the processing and block the execution (TBD). The source can be a class that naturally produces data when polled (by blocking) or an iterable of data.
+
+
+## Example syntax
+### Defaults
+```python
+pipe = Pipeline()
+pipe.add_source(UserInputText())
+pipe.add_step(WeatherForecast())
+pipe.add_step(print)
+pipe.add_sink(DBStore())
+pipe.process()
+```
+
+### Adapter functions
+```python
+pipe = Pipeline()
+pipe.add_source(UserInputText(), map = lambda x: x.strip())
+pipe.add_step(WeatherForecast(), map = lambda x: x.temperature_c, max_rate = "1/s")
+pipe.add_step(print)
+pipe.add_sink(DBStore(), map = lambda x: { "temperature_c": x.temperature_c, "time": datetime.now() })
+pipe.process()
+```

design/imperative.md

@@ -0,0 +1,20 @@
+# Imperative style design
+
+## Requirements
+
+1. Imperative programming style
+2. Simple and straightforward: no need to adapt to a framework, the user "just writes" code in the most basic way
+
+## Example syntax
+```python
+userinput = UserTextInput()
+city = userinput.get()
+
+weather_forecast = WeatherForecast()
+forecast = weather_forecast.get_forecast_by_city(city)
+
+print(forecast.temperature_c)
+
+db = DBStore()
+db.save(forecast.temperature_c)
+```