From 86abca9e308def2f1e4ac92b03580d0f03a9c28a Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 3 Nov 2025 18:47:02 +0000 Subject: [PATCH 1/9] Initial plan From 08177a573d9e5278f53ff9c1b49f08abeb63d9d8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 3 Nov 2025 18:57:22 +0000 Subject: [PATCH 2/9] Add Blockly integration demo for OT-2 custom functions Co-authored-by: sgbaird <45469701+sgbaird@users.noreply.github.com> --- .gitignore | 1 + CHANGELOG.md | 6 + scripts/BLOCKLY_README.md | 161 +++++++++++++ scripts/blockly_concept_demo.html | 363 ++++++++++++++++++++++++++++++ scripts/blockly_concept_demo.png | Bin 0 -> 378857 bytes scripts/blockly_example.py | 186 +++++++++++++++ scripts/blockly_ot2_demo.html | 343 ++++++++++++++++++++++++++++ 7 files changed, 1060 insertions(+) create mode 100644 scripts/BLOCKLY_README.md create mode 100644 scripts/blockly_concept_demo.html create mode 100644 scripts/blockly_concept_demo.png create mode 100644 scripts/blockly_example.py create mode 100644 scripts/blockly_ot2_demo.html diff --git a/.gitignore b/.gitignore index 52cd3236..e26b00e5 100644 --- a/.gitignore +++ b/.gitignore @@ -72,3 +72,4 @@ src/ac_training_lab/apriltag_demo/tag25_09_00000.png src/ac_training_lab/apriltag_demo/tag36_11_00000.png src/ac_training_lab/apriltag_demo/tag49_12_00000.png scripts/playwright/sem-open-close/chat.json +scripts/blockly_demo/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d8cd7b1..a3519da4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,12 @@ ## [Unreleased] ### Added - Support for both `rpicam-vid` (Raspberry Pi OS Trixie) and `libcamera-vid` (Raspberry Pi OS Bookworm) camera commands in `src/ac_training_lab/picam/device.py` to ensure compatibility across different OS versions. +- Blockly integration demonstration showing how custom visual programming blocks can be created for OT-2 functions from `OT2mqtt.py`: + - `scripts/blockly_concept_demo.html`: Visual demonstration of block-to-code mapping + - `scripts/blockly_ot2_demo.html`: Interactive Blockly editor with custom OT-2 blocks + - `scripts/blockly_example.py`: Python example showing generated code patterns + - `scripts/BLOCKLY_README.md`: Comprehensive documentation for Blockly integration + - Custom blocks for `mix_color()`, `move_sensor_back()`, and `protocol.home()` functions ### Fixed - Ctrl+C interrupt handling in `src/ac_training_lab/picam/device.py` now properly exits the streaming loop instead of restarting. diff --git a/scripts/BLOCKLY_README.md b/scripts/BLOCKLY_README.md new file mode 100644 index 00000000..82866dc1 --- /dev/null +++ b/scripts/BLOCKLY_README.md @@ -0,0 +1,161 @@ +# Blockly OT-2 Integration Demo + +This directory contains a demonstration of how [Blockly](https://developers.google.com/blockly) can be integrated with custom functions from the AC Training Lab, specifically the OT-2 liquid handler functions from `OT2mqtt.py`. + +## Overview + +Blockly is a visual programming library that allows users to create workflows by dragging and dropping blocks. This demo shows how to: + +1. Create custom Blockly blocks that represent OT-2 functions +2. Generate Python code from visual blocks that can call functions from `OT2mqtt.py` +3. Provide a one-to-one correspondence between visual programming and Python code + +## What's Included + +### `blockly_ot2_demo.html` + +An interactive HTML page that demonstrates: + +- **Custom OT-2 Blocks**: Blocks for key functions from `OT2mqtt.py`: + - `mix_color`: Mix RGB colors using the OT-2 pipette + - `move_sensor_back`: Return sensor to charging position + - `ot2_home`: Home the OT-2 robot + +- **Standard Blockly Blocks**: Logic, loops, math, text, and variables for building complex workflows + +- **Real-time Code Generation**: As you modify the blocks, the corresponding Python code is generated instantly + +## How to Use + +### Option 1: Direct Browser Access + +1. Simply open `blockly_ot2_demo.html` in a web browser: + ```bash + # From the scripts directory + firefox blockly_ot2_demo.html + # or + google-chrome blockly_ot2_demo.html + # or + open blockly_ot2_demo.html # macOS + ``` + +2. The page loads Blockly from a CDN, so no installation is required + +### Option 2: Using Python HTTP Server + +If you need to serve the file over HTTP: + +```bash +# From the repository root +cd scripts +python -m http.server 8000 +``` + +Then navigate to `http://localhost:8000/blockly_ot2_demo.html` in your browser. + +## Example Workflow + +The demo includes a pre-loaded example that demonstrates: + +1. Homing the OT-2 robot +2. Repeating 3 times: + - Mixing colors (100µL red, 50µL yellow, 75µL blue) in well A1 + - Moving the sensor back to charging position + +This generates Python code like: + +```python +# Generated from Blockly visual programming +# This code uses functions from OT2mqtt.py + +from OT2mqtt import mix_color, move_sensor_back, protocol + +# Main workflow +protocol.home() +for count in range(3): + payload = { + "command": { + "R": 100, + "Y": 50, + "B": 75, + "well": "A1" + }, + "session_id": "session_001", + "experiment_id": "exp_001" + } + mix_color(payload) + payload = { + "command": { + "sensor_status": "complete" + }, + "session_id": "session_001", + "experiment_id": "exp_001" + } + move_sensor_back(payload) +``` + +## Extending with More Functions + +To add more custom functions from `OT2mqtt.py` or other modules: + +1. **Define the Block**: Add a new block definition in the JavaScript: + ```javascript + Blockly.Blocks['your_block_name'] = { + init: function() { + this.appendDummyInput() + .appendField("Your Block Label"); + this.appendValueInput("PARAM1") + .setCheck("Number") + .appendField("Parameter 1"); + // Add more inputs as needed + this.setPreviousStatement(true, null); + this.setNextStatement(true, null); + this.setColour(230); + this.setTooltip("Description"); + } + }; + ``` + +2. **Define Code Generation**: Add Python code generator: + ```javascript + Blockly.Python['your_block_name'] = function(block) { + var value_param1 = Blockly.Python.valueToCode(block, 'PARAM1', Blockly.Python.ORDER_ATOMIC); + var code = `your_function(${value_param1})\n`; + return code; + }; + ``` + +3. **Add to Toolbox**: Include the block in the toolbox XML + +## Key Features Demonstrated + +1. **Visual to Code**: One-to-one correspondence between blocks and Python code +2. **Custom Functions**: Integration with existing Python functions +3. **Composability**: Combine custom blocks with standard control flow blocks +4. **Real-time Feedback**: See generated code update as you modify blocks +5. **No Installation**: Runs directly in the browser using CDN-hosted Blockly + +## Comparison with Other Tools + +Like n8n, NODE-RED, and IvoryOS, Blockly provides a visual drag-and-drop interface. However, Blockly's key advantage is the direct Python code generation, making it ideal for: + +- Educational purposes (learn programming visually, then see the code) +- Rapid prototyping of lab workflows +- Making automation accessible to non-programmers +- Generating reproducible Python scripts from visual designs + +## Next Steps + +To use this in a production environment: + +1. Create a backend service that receives the generated Python code +2. Validate and sanitize the code before execution +3. Execute in a controlled environment with appropriate error handling +4. Add authentication and authorization +5. Integrate with existing MQTT broker for OT-2 communication + +## References + +- [Blockly Official Documentation](https://developers.google.com/blockly) +- [Blockly Getting Started Guide](https://blocklycodelabs.dev/codelabs/getting-started/index.html) +- [OT-2 Python API](https://docs.opentrons.com/v2/) diff --git a/scripts/blockly_concept_demo.html b/scripts/blockly_concept_demo.html new file mode 100644 index 00000000..3fbb35df --- /dev/null +++ b/scripts/blockly_concept_demo.html @@ -0,0 +1,363 @@ + + +
+ +Visual Programming for Laboratory Automation
+# Generated from Blockly visual programming
+# This code uses functions from OT2mqtt.py
+
+from OT2mqtt import mix_color, move_sensor_back, protocol
+
+# Main workflow
+protocol.home()
+
+for count in range(3):
+ payload = {
+ "command": {
+ "R": 100,
+ "Y": 50,
+ "B": 75,
+ "well": "A1"
+ },
+ "session_id": "session_001",
+ "experiment_id": "exp_001"
+ }
+ mix_color(payload)
+
+ payload = {
+ "command": {
+ "sensor_status": "complete"
+ },
+ "session_id": "session_001",
+ "experiment_id": "exp_001"
+ }
+ move_sensor_back(payload)
+ Drag-and-drop interface makes it easy for non-programmers to create complex lab workflows
+One-to-one correspondence between visual blocks and Python code ensures predictability
+Users can learn Python by seeing how their visual programs translate to code
+Easy to create custom blocks for any function in OT2mqtt.py or other modules
+Generated code can be saved, version controlled, and executed in production environments
+Visual workflows ensure consistent execution and make it easy to share protocols
+)tul84DkqRLyK$+0X-BT9
zE#ts$UOM$!zEXkPI5h u7LQf7H#(
zkO$s@fuEQYdZ%I*_9}F6mHJ^}T39lyh#K}Td%|N_eCkH1Dp)9mT9xh;u)h`03jq13
zpFT`Xj@^v#>HVUZaTHl^eD6321Wl$$d%US{$pS`w;iS+-f;q>E(s}B0vub4U1q}Y)
zTa-Kk*uU5QT#)I+gRCUa{2C-?KR0ktCGYatHZ1Y$)&oM`lyt)vAQb;B!FdIIS8HyW
zUYkHwqH+y^ZQ0#mQGJBf=MAb-8_xbJ(VSlO5O*62@^@3CfH5eRez;AiXvKp+p5H6h
z-^XgnQk)kTwnTiX0|qkV0S^cpbo$>QxEV#$sYvrQz~&H-NG)-VHXw(gDyTU=^>H0F
zt43T>jVPyTvJ59$B7RX*!4;=cRW+*A-n#_dZRWr|GBIex6Hz7|l}5D-ef-XHy2XJT
zN@ojdE+yE8R&Mh8-kZ1*0!+JVyvUSbEM>6Q&@D+P(RZg$7LCJFZmY%9J4rivRvX{_
zb@xucGi~G(9is7Fy2O~*$K6#9c{>LUUy=IIv-2qL(A(TDLBi9FukT)I?m!IX-=+`%
zz{fQvsK6vnr3Du
zBU{YOzdL#WmaZ6#QgbQ$x7?gQX5Ja*URJ#=7Kr!
z?zH{4m26JFgh?=F>IsI9xYstHjmd`gZfA8Cg?JfkkIPFuA;F@v_rN-{?+q#uO-C-bi
z7>FONQWo5esd&}xwuqy8aEj8kwJRHNJd}bRt7g?|LYsnG`+NnyP{GvIirLNHDDI%e
zN)uZ-Bv&8*NOTmCJN9s7RFF91%*{P}+~RH}S=9@3DX~S%)QX=5qXlbaMTCI61esQX
zSlokZDlsBghEz(7FNjsTnTm4_`P7dfr5&yE4k#{qS6|hkIb|j1M9jKgPYUv$>=@=R
zU2DX?RR!hqOwDbw#gcJxu9W5`BsBcqtbrO<2EEa$%xPp+>k>GOcxIk_ByPQFGei|I
z7IOODfQ#lDH5;E2=$12~fJ;|fZN@nJu{DR0s~g+DmsKR1)h!ZltA8XL2?+1dH%|VT
z9_?n>&DbsY8Ly8(cL=gmBC#%A=6*J{CI-Y~N5pJGWe?ru^8uR3ixP+1;%YfUV@IRC
z2mzVGfs_ViBPHxhG1_LGVbe1dr`IMRsahw_4oclZ>?fK&SYhc4AaPy{Ct9-*rNr#(
z0nX_hy3&jgv6sb2SLx