Skip to content

Commit 9b99e7c

Browse files
committed
AP-25351: Add copilot instructions
AP-25351 (Add GitHub Actions for repositories containing scripting editors)
1 parent 7b0916f commit 9b99e7c

File tree

4 files changed

+137
-0
lines changed

4 files changed

+137
-0
lines changed

.github/copilot-instructions.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# KNIME Python Integration
2+
3+
Hybrid Java/Python/JavaScript codebase bridging KNIME Analytics Platform with Python via Py4J and Apache Arrow.
4+
5+
## Architecture Overview
6+
7+
**Components**:
8+
- `org.knime.python3*`: OSGi plugins (Eclipse bundles with `META-INF/MANIFEST.MF`)
9+
- `org.knime.python3.scripting.nodes/js-src/`: Vue 3 scripting editor
10+
- `knime-extension/`: Conda package for Python node development
11+
12+
**Java-Python Bridge**: Py4J enables bidirectional communication. Java spawns Python processes via `PythonGateway`; Python extends `EntryPoint` classes to expose functionality.
13+
14+
**Data Transfer**: Apache Arrow IPC format via memory-mapped files. Tables serialized through `PythonArrowDataSource`/`PythonArrowDataSink`, metadata (types/domain) passed as JSON.
15+
16+
**Node Extension Flow**:
17+
1. Python defines nodes with `@knext.node`/`@knext.parameter` decorators
18+
2. Java's `PurePythonNodeSetFactory` discovers them via Py4J at KNIME startup
19+
3. `DelegatingNodeModel` proxies execution to Python via `NodeProxy` interfaces
20+
4. Data flows through Arrow, callbacks through Py4J (e.g., `LogCallback`, `AuthCallback`)
21+
22+
## Module Structure
23+
24+
```
25+
org.knime.python3 # Core Py4J gateway
26+
org.knime.python3.arrow # Arrow data transfer (Java)
27+
org.knime.python3.arrow.types # KNIME types (Python)
28+
org.knime.python3.nodes # Node framework (Java)
29+
org.knime.python3.scripting # Scripting infrastructure
30+
org.knime.python3.scripting.nodes # Script nodes + Vue editor
31+
org.knime.python3.views # Python view rendering
32+
org.knime.ext.py4j # Py4J OSGi wrapper
33+
*.py4j.dependencies # Expose classes to py4j classloader (don't merge!)
34+
```
35+
36+
## Quick Start
37+
38+
```bash
39+
pixi install # Setup Python env (.pixi/envs/default)
40+
pixi run test-all # Python tests (interactive env)
41+
mvn clean verify # Java tests
42+
cd org.knime.python3.scripting.nodes/js-src && npm ci && npm run coverage # JS tests
43+
```
44+
45+
## CI/CD
46+
47+
- **Jenkinsfile**: Primary CI
48+
- **GitHub Actions**: Fast pre-checks: lint, test, build JS, format Python
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
applyTo: "**/*.java"
3+
---
4+
# Java Development - KNIME Python Integration
5+
6+
## Build System (Tycho, NOT Standard Maven)
7+
```bash
8+
mvn clean verify # Full build + tests (Tycho 4.0.13 in .mvn/extensions.xml)
9+
```
10+
11+
**P2 Dependencies** (root `pom.xml`): Dependencies from Eclipse P2 repos, NOT Maven Central.
12+
13+
## Repository-Specific Conventions
14+
- **OSGi bundles**: Each `org.knime.*` folder has `META-INF/MANIFEST.MF` + `build.properties`
15+
- **Test bundles**: Name ends `.tests`, use `eclipse-test-plugin` packaging
16+
- **Test POMs**: Require `org.knime.features.clfixes`, Arrow tests need `org.knime.features.core.columnar`
17+
- **Critical**: `*.py4j.dependencies` bundles expose classes to py4j classloader - **don't merge!**
18+
19+
## Key Classes (Repository-Specific)
20+
- `PythonGateway`: Spawns Python processes
21+
- `PythonArrowDataSource`/`Sink`: Arrow I/O with Python (temp files, metadata as JSON)
22+
- `DelegatingNodeModel`: Proxies node execution to Python
23+
- `PurePythonNodeSetFactory`: Discovers Python nodes via Py4J at startup
24+
- `QueuedPythonGatewayFactory`: Gateway lifecycle singleton
25+
26+
## Common Pitfalls
27+
1. **P2 only**: Can't use Maven Central deps
28+
2. **OSGi classloading**: Keep `*.py4j.dependencies` separate
29+
3. **Test failures**: Missing `org.knime.features.clfixes` in test POM
30+
4. **Bundle exports**: Explicitly list in `MANIFEST.MF`
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
applyTo: "org.knime.python3.scripting.nodes/js-src/**"
3+
---
4+
# JavaScript/TypeScript Development - KNIME Python Integration
5+
6+
## Location
7+
`org.knime.python3.scripting.nodes/js-src/` - Python scripting editor dialog
8+
9+
## Setup
10+
```bash
11+
cd org.knime.python3.scripting.nodes/js-src
12+
npm ci # Node 22.11.0, npm 10.9.1
13+
npm run coverage # Tests + coverage (do not use `npm run test:unit` - it's interactive)
14+
```
15+
16+
## Repository-Specific Conventions
17+
- **Maven integration**: `frontend-maven-plugin` runs npm during Maven build
18+
- **Dependency**: Uses `@knime/scripting-editor` - unlink before committing if testing local changes
19+
20+
## KNIME Dev Mode
21+
Start KNIME with:
22+
```
23+
-Dchromium.remote_debugging_port=8988
24+
-Dorg.knime.ui.dev.node.dialog.url=http://localhost:5173/
25+
```
26+
Then: `npm run dev:knime`
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
applyTo: "**/*.py"
3+
---
4+
# Python Development - KNIME Python Integration
5+
6+
## Dependencies & Testing
7+
```bash
8+
pixi install # Setup .pixi/envs/default
9+
pixi run test-all # Test across py38-py314, KNIME 4.7-5.6
10+
pixi run format # Ruff (target: Python 3.8)
11+
```
12+
13+
**All dependencies via `pixi.toml`** - never pip/conda directly.
14+
15+
## Repository-Specific Conventions
16+
- **Path setup**: Root `conftest.py` adds all plugin paths to `sys.path` - **never modify `sys.path` in tests**
17+
- **Test style**: Function-based tests only (`pytest.ini` disables class discovery)
18+
- **Multi-version**: Changes must work across Python 3.8-3.14 - test with `pixi run test-all`
19+
- **PyArrow versions**: Tightly coupled to Python version in `pixi.toml` - don't update independently
20+
21+
## Key Modules (Repository-Specific)
22+
- `knext`: Node extension decorators (`@knext.node`, `@knext.parameter`)
23+
- `knime.scripting`: Script API (replaces deprecated `knime_io`, `knime_jupyter`)
24+
- `knime._arrow._backend`: Arrow I/O with Java (via Py4J)
25+
26+
## Py4J Integration
27+
- Extend Java `EntryPoint` classes to expose Python to Java
28+
- Implement Java interfaces for callbacks (`LogCallback`, `AuthCallback`)
29+
- Arrow data via temp files (IPC format), metadata as JSON
30+
31+
## Debugging
32+
- `KNIME_PYTHON_DEBUG=true` for gateway logs
33+
- Import failures: check `conftest.py`

0 commit comments

Comments
 (0)