|
| 1 | +# KNIME Python Integration |
| 2 | + |
| 3 | +Hybrid Java/Python/JavaScript codebase bridging KNIME Analytics Platform with Python via Py4J and Apache Arrow. |
| 4 | + |
| 5 | +## Architecture Overview |
| 6 | + |
| 7 | +**Components**: |
| 8 | +- `org.knime.python3*`: OSGi plugins (Eclipse bundles with `META-INF/MANIFEST.MF`) |
| 9 | +- `org.knime.python3.scripting.nodes/js-src/`: Vue 3 scripting editor |
| 10 | +- `knime-extension/`: Conda package for Python node development |
| 11 | + |
| 12 | +**Java-Python Bridge**: Py4J enables bidirectional communication. Java spawns Python processes via `PythonGateway`; Python extends `EntryPoint` classes to expose functionality. |
| 13 | + |
| 14 | +**Data Transfer**: Apache Arrow IPC format via memory-mapped files. Tables serialized through `PythonArrowDataSource`/`PythonArrowDataSink`, metadata (types/domain) passed as JSON. |
| 15 | + |
| 16 | +**Node Extension Flow**: |
| 17 | +1. Python defines nodes with `@knext.node`/`@knext.parameter` decorators |
| 18 | +2. Java's `PurePythonNodeSetFactory` discovers them via Py4J at KNIME startup |
| 19 | +3. `DelegatingNodeModel` proxies execution to Python via `NodeProxy` interfaces |
| 20 | +4. Data flows through Arrow, callbacks through Py4J (e.g., `LogCallback`, `AuthCallback`) |
| 21 | + |
| 22 | +## Module Structure |
| 23 | + |
| 24 | +``` |
| 25 | +org.knime.python3 # Core Py4J gateway |
| 26 | +org.knime.python3.arrow # Arrow data transfer (Java) |
| 27 | +org.knime.python3.arrow.types # KNIME types (Python) |
| 28 | +org.knime.python3.nodes # Node framework (Java) |
| 29 | +org.knime.python3.scripting # Scripting infrastructure |
| 30 | +org.knime.python3.scripting.nodes # Script nodes + Vue editor |
| 31 | +org.knime.python3.views # Python view rendering |
| 32 | +org.knime.ext.py4j # Py4J OSGi wrapper |
| 33 | +*.py4j.dependencies # Expose classes to py4j classloader (don't merge!) |
| 34 | +``` |
| 35 | + |
| 36 | +## Quick Start |
| 37 | + |
| 38 | +```bash |
| 39 | +pixi install # Setup Python env (.pixi/envs/default) |
| 40 | +pixi run test-all # Python tests (interactive env) |
| 41 | +mvn clean verify # Java tests |
| 42 | +cd org.knime.python3.scripting.nodes/js-src && npm ci && npm run coverage # JS tests |
| 43 | +``` |
| 44 | + |
| 45 | +## CI/CD |
| 46 | + |
| 47 | +- **Jenkinsfile**: Primary CI |
| 48 | +- **GitHub Actions**: Fast pre-checks: lint, test, build JS, format Python |
0 commit comments