From 5e6da13b14270bb30ac2920082d3e2146529550d Mon Sep 17 00:00:00 2001 From: Jinkun Liu Date: Sun, 24 May 2026 16:50:17 +0800 Subject: [PATCH 1/4] [docs] Update installation instructions for Flink Agents --- docs/content/docs/get-started/installation.md | 191 ++++++++++-------- 1 file changed, 105 insertions(+), 86 deletions(-) diff --git a/docs/content/docs/get-started/installation.md b/docs/content/docs/get-started/installation.md index 4a0c8d989..388810a43 100644 --- a/docs/content/docs/get-started/installation.md +++ b/docs/content/docs/get-started/installation.md @@ -31,7 +31,56 @@ The sections below show how to install the required dependencies. __NOTE:__ To run on a Flink cluster, Flink-Agents requires a stable release of Flink version 1.20.3 or higher. We highly recommend using the **latest stable release of your chosen Flink minor version** (e.g., for minor version 2.2, use the latest 2.2.x release). {{< /hint >}} -## Install Apache Flink +## Prerequisites + +Both the script-based and manual installation paths require: + +* **Java 11+** on your `PATH`. Java 21+ is recommended when using the Java API — see the table below for the version-specific limitations. +* **Python 3.10, 3.11, or 3.12** (only required if you plan to use the Python API or PyFlink). + +For building Flink Agents from source, you additionally need: +- Unix-like environment (Linux, macOS, Cygwin, or WSL) +- Git +- Maven 3 + +### Java Versions + +For running an agent built with the **Python API**, you can use any Java version 11 or higher. + +When using the **Java API**, there are some functionality limitations for earlier Java versions, as detailed below: + +| Java Version | Limitations | +|--------------|------------------------| +| Java 21+ | No limitations. | +| Java 11-20 | Async execution is unavailable. | + +{{< hint info >}} +**Note**: If you want to use python 3.12, you need install Flink above 2.1 (including 2.1). +{{< /hint >}} + +## Quick install (recommended) + +The `install.sh` script provisions everything you need: it downloads Apache Flink, creates a Python virtual environment, installs `flink-agents` and `apache-flink` into it, and copies the required JARs into `$FLINK_HOME/lib`. + +Run the one-liner from any directory you want the virtual environment created in: + +```shell +curl -fsSL https://raw.githubusercontent.com/apache/flink-agents/main/tools/install.sh | bash +``` + +### What the script does + +- Verifies that Java 11+ is available on your `PATH`. It does **not** install Java for you. +- Downloads and extracts Apache Flink into `$INSTALL_DIR` (default: `$HOME/.local/flink`), or reuses an existing `$FLINK_HOME` if you tell it not to install Flink. +- Creates a Python virtual environment (default: `./.flink-agents-env`) and runs `pip install flink-agents apache-flink` inside it. +- Copies `flink-python-.jar`, `flink-agents-dist-common-*.jar`, and the version-specific `flink-agents-dist-*` thin JAR into `$FLINK_HOME/lib`. +- Runs post-install verification (Flink binary, Python package, and JAR presence) before exiting. + +## Manual installation + +Use this path when you want full control over each step, when running `install.sh` is not feasible in your environment, or when building Flink Agents from source. + +### Install Apache Flink Before installing Flink Agents, you need to have Apache Flink installed. @@ -56,32 +105,6 @@ cp $FLINK_HOME/opt/flink-python-${FLINK_VERSION}.jar $FLINK_HOME/lib/ **Note:** For more detailed Flink installation instructions, refer to the [Flink local installation guide](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/try-flink/local_installation/). {{< /hint >}} -{{< hint info >}} -**Note**: If you want to use python 3.12, you need install Flink above 2.1 (including 2.1). -{{< /hint >}} - -## Install Flink Agents - -### Prerequisites - -* Python 3.10, 3.11 or 3.12 -* For building from source, you also need: - - Unix-like environment (Linux, Mac OS X, Cygwin, or WSL) - - Git - - Maven 3 - - Java 21+ (full functionality), or Java 11+ (some features unavailable) - -### Java Versions - -For running an agent built with **Python API**, you can use any Java version 11 or higher. - -When using **Java API**, there are some functionality limitations for earlier Java versions, as detailed below: - -| Java Version | Limitations | -|--------------|------------------------| -| Java 21+ | No limitations. | -| Java 11-20 | Async execution is unavailable. | - ### Set Up Python Environment (Recommended) We recommend using a Python virtual environment to isolate Flink Agents dependencies from your system Python packages. @@ -158,6 +181,60 @@ After building: - The Python package is installed and ready to use - The distribution JAR is located at: `dist/flink-${FLINK_VERSION%.*}/target/flink-agents-dist-*.jar` +### Configure PYTHONPATH + +Flink runs in its own JVM process and needs the `PYTHONPATH` environment variable to locate the flink-agents Python package. You need to set `PYTHONPATH` to the directory where flink-agents is installed. + +**Determine your Python package installation path:** + +The path depends on your Python environment setup: +- If using a virtual environment, it's the site-packages directory within your venv +- If using system Python, it's the system site-packages directory + +**Tip:** You can use this command to help find the path: +```shell +python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' +``` + +**Set PYTHONPATH before starting Flink:** + +```shell +# Set PYTHONPATH to your Python site-packages directory + +export PYTHONPATH=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])') +``` + +{{< hint info >}} +**Note:** You can add the `export PYTHONPATH=...` line to your shell profile (`~/.bash_profile`, `~/.bashrc`, `~/.zprofile`, or `~/.zshrc`) to set it permanently. This way, it will be automatically configured in all future terminal sessions. +{{< /hint >}} + +### Install Flink Agents Java Library + +Copy the Flink Agents distribution JAR to your Flink installation's `lib` directory: + +{{< tabs "Install Flink Agents Java Library" >}} + +{{< tab "From Official Release" >}} +The Flink Agents JAR is bundled inside the Python package. Use the PYTHONPATH you configured above to locate and copy it: + +```shell +# Copy the JAR from the Python package to Flink's lib directory +cp $PYTHONPATH/flink_agents/lib/flink-${FLINK_VERSION%.*}/flink-agents-dist-*.jar $FLINK_HOME/lib/ +``` + +{{< /tab >}} + +{{< tab "From Source" >}} +After building from source, the distribution JAR is located in the `dist/target/` directory: + +```shell +# Copy the JAR to Flink's lib directory +cp dist/flink-${FLINK_VERSION%.*}/target/flink-agents-dist-*.jar $FLINK_HOME/lib/ +``` +{{< /tab >}} + +{{< /tabs >}} + ### Build Environment Variables The following environment variables can be used to control how JARs are resolved during `pip install flink-agents` (from sdist) or `python -m build`: @@ -229,67 +306,9 @@ For execution in IDE, enable the feature `add dependencies with provided scope t ## Deploy to Flink Cluster -After installing Flink Agents package, you need to deploy it to your Flink cluster so that Flink can run your agent jobs. - -### Configure PYTHONPATH - -Flink runs in its own JVM process and needs the `PYTHONPATH` environment variable to locate the flink-agents Python package. You need to set `PYTHONPATH` to the directory where flink-agents is installed. - -**Determine your Python package installation path:** - -The path depends on your Python environment setup: -- If using a virtual environment, it's the site-packages directory within your venv -- If using system Python, it's the system site-packages directory - -**Tip:** You can use this command to help find the path: -```shell -python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' -``` - -**Set PYTHONPATH before starting Flink:** - -```shell -# Set PYTHONPATH to your Python site-packages directory - -export PYTHONPATH=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])') -``` - -{{< hint info >}} -**Note:** You can add the `export PYTHONPATH=...` line to your shell profile (`~/.bash_profile`, `~/.bashrc`, `~/.zprofile`, or `~/.zshrc`) to set it permanently. This way, it will be automatically configured in all future terminal sessions. -{{< /hint >}} - -### Install Flink Agents Java Library - -Copy the Flink Agents distribution JAR to your Flink installation's `lib` directory: - -{{< tabs "Install Flink Agents Java Library" >}} - -{{< tab "From Official Release" >}} -The Flink Agents JAR is bundled inside the Python package. Use the PYTHONPATH you configured above to locate and copy it: - -```shell -# Copy the JAR from the Python package to Flink's lib directory -cp $PYTHONPATH/flink_agents/lib/flink-${FLINK_VERSION%.*}/flink-agents-dist-*.jar $FLINK_HOME/lib/ -``` - -{{< /tab >}} - -{{< tab "From Source" >}} -After building from source, the distribution JAR is located in the `dist/target/` directory: - -```shell -# Copy the JAR to Flink's lib directory -cp dist/flink-${FLINK_VERSION%.*}/target/flink-agents-dist-*.jar $FLINK_HOME/lib/ -``` -{{< /tab >}} - -{{< /tabs >}} - -### Start Flink Cluster or Submit Job - Once the Java library is installed and PYTHONPATH is configured, you can start your Flink cluster or submit jobs: -{{< tabs "Start Flink Cluster or Submit Job" >}} +{{< tabs "Deploy to Flink Cluster" >}} {{< tab "Python" >}} ```bash From 103d07a62b55380034a3ca133bc614740fd658c1 Mon Sep 17 00:00:00 2001 From: Jinkun Liu Date: Sun, 24 May 2026 16:51:13 +0800 Subject: [PATCH 2/4] [tools] Remove post-install verification option and fix the verification bug --- tools/install.sh | 51 +++++++++++++-------------------- tools/test/helpers/load.bash | 1 - tools/test/unit/parse_args.bats | 8 +----- 3 files changed, 21 insertions(+), 39 deletions(-) diff --git a/tools/install.sh b/tools/install.sh index c71e3fa31..5121ebd9f 100755 --- a/tools/install.sh +++ b/tools/install.sh @@ -473,7 +473,6 @@ PYTHON_BIN="${PYTHON_BIN:-}" NO_PROMPT="${NO_PROMPT:-0}" VERBOSE="${FLINK_AGENTS_VERBOSE:-0}" DRY_RUN="${FLINK_AGENTS_DRY_RUN:-0}" -VERIFY_INSTALL="${FLINK_AGENTS_VERIFY_INSTALL:-0}" HELP=0 PYFLINK_ACTUALLY_ENABLED=0 @@ -491,7 +490,6 @@ Options: --enable-pyflink Enable PyFlink and install Python packages --verbose Print debug output (set -x) --dry-run Print install plan without making changes - --verify Run post-install verification checks --python Path to a Python3 interpreter (overrides PATH lookup) --flink-version Apache Flink version (e.g. 2.2.0); overrides the interactive picker --flink-agents-version @@ -513,7 +511,6 @@ Environment variables: NO_PROMPT 1 to disable all prompts FLINK_AGENTS_VERBOSE 1 to enable verbose output FLINK_AGENTS_DRY_RUN 1 to enable dry-run mode - FLINK_AGENTS_VERIFY_INSTALL 1 to enable post-install verification Examples: bash install.sh --install-flink --enable-pyflink --non-interactive @@ -1670,17 +1667,9 @@ show_install_plan() { if [[ "$DRY_RUN" == "1" ]]; then ui_kv "Dry run" "yes" fi - if [[ "$VERIFY_INSTALL" == "1" ]]; then - ui_kv "Verify" "yes" - fi } verify_installation() { - if [[ "${VERIFY_INSTALL}" != "1" ]]; then - return 0 - fi - - ui_stage "Verifying installation" if [[ -x "$FLINK_HOME/bin/flink" ]]; then local flink_ver @@ -1703,7 +1692,7 @@ verify_installation() { fi if [[ "$PYFLINK_ACTUALLY_ENABLED" -eq 1 ]]; then - if python -c "import flink_agents; print('flink-agents', flink_agents.__version__)" 2>/dev/null; then + if "$VENV_DIR/bin/python3" -c "import flink_agents; print('flink-agents ready')" 2>/dev/null; then ui_success "flink-agents Python package verified" else ui_error "flink-agents Python package import failed" @@ -1720,6 +1709,24 @@ verify_installation() { ui_success "Verification complete" } +show_next_step(){ + echo "" + ui_celebrate "Apache Flink Agents installation finished!" + echo "" + ui_section "Next steps" + ui_success "1) Point FLINK_HOME at this install (Flink CLI and clients read it):" + ui_info " export FLINK_HOME=${FLINK_HOME}" + if [[ "$PYFLINK_ACTUALLY_ENABLED" -eq 1 ]]; then + ui_success "2) Activate the Python venv (PyFlink + flink-agents are installed there):" + ui_info " source ${VENV_DIR}/bin/activate" + ui_success "3) To make both permanent, append the two lines above to your shell rc" + ui_info " (~/.zshrc or ~/.bashrc)." + else + ui_success "2) To make it permanent, append the line above to your shell rc" + ui_info " (~/.zshrc or ~/.bashrc)." + fi +} + show_footer_links() { local docs_url="https://nightlies.apache.org/flink/flink-agents-docs-latest/" local issues_url="https://github.com/apache/flink-agents/issues" @@ -1751,10 +1758,6 @@ parse_args() { DRY_RUN=1 shift ;; - --verify) - VERIFY_INSTALL=1 - shift - ;; --python) if [[ $# -lt 2 ]]; then die "--python requires a path argument" @@ -1859,21 +1862,7 @@ main() { verify_installation - echo "" - ui_celebrate "Apache Flink Agents installation finished!" - echo "" - ui_section "Next steps" - ui_success "1) Point FLINK_HOME at this install (Flink CLI and clients read it):" - ui_info " export FLINK_HOME=${FLINK_HOME}" - if [[ "$PYFLINK_ACTUALLY_ENABLED" -eq 1 ]]; then - ui_success "2) Activate the Python venv (PyFlink + flink-agents are installed there):" - ui_info " source ${VENV_DIR}/bin/activate" - ui_success "3) To make both permanent, append the two lines above to your shell rc" - ui_info " (~/.zshrc or ~/.bashrc)." - else - ui_success "2) To make it permanent, append the line above to your shell rc" - ui_info " (~/.zshrc or ~/.bashrc)." - fi + show_next_step show_footer_links } diff --git a/tools/test/helpers/load.bash b/tools/test/helpers/load.bash index 308d450da..043a4fd67 100644 --- a/tools/test/helpers/load.bash +++ b/tools/test/helpers/load.bash @@ -24,7 +24,6 @@ reset_install_sh_state() { DRY_RUN=0 NO_PROMPT=0 VERBOSE=0 - VERIFY_INSTALL=0 INSTALL_FLINK="Ask" ENABLE_PYFLINK="Ask" PYTHON_BIN="" diff --git a/tools/test/unit/parse_args.bats b/tools/test/unit/parse_args.bats index 76084472f..62b6b252e 100644 --- a/tools/test/unit/parse_args.bats +++ b/tools/test/unit/parse_args.bats @@ -36,11 +36,6 @@ setup() { [ "$DRY_RUN" = "1" ] } -@test "parse_args: --verify sets VERIFY_INSTALL=1" { - parse_args --verify - [ "$VERIFY_INSTALL" = "1" ] -} - @test "parse_args: --python sets PYTHON_BIN" { parse_args --python /opt/py/bin/python3 [ "$PYTHON_BIN" = "/opt/py/bin/python3" ] @@ -74,9 +69,8 @@ setup() { } @test "parse_args: combined flags all apply" { - parse_args --non-interactive --install-flink --enable-pyflink --verify + parse_args --non-interactive --install-flink --enable-pyflink [ "$NO_PROMPT" = "1" ] [ "$INSTALL_FLINK" = "Yes" ] [ "$ENABLE_PYFLINK" = "Yes" ] - [ "$VERIFY_INSTALL" = "1" ] } From db163e8b984da505cfc2d20e8efd87dbed6d159f Mon Sep 17 00:00:00 2001 From: Jinkun Liu Date: Sun, 24 May 2026 18:07:26 +0800 Subject: [PATCH 3/4] [tools] Reattach stdin to /dev/tty in curl|bash installs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the installer is invoked via `curl … | bash`, stdin is the pipe, so is_non_interactive_shell short-circuits gum bootstrap and every interactive prompt falls back to plain bash menus that can't read user input either. Reattach stdin to the controlling terminal at the top of main(), guarded by NO_PROMPT and a real /dev/tty open test so CI and no-TTY containers continue to use the silent fallback. Co-Authored-By: Claude Opus 4.7 (1M context) --- tools/install.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/install.sh b/tools/install.sh index 5121ebd9f..f43499c05 100755 --- a/tools/install.sh +++ b/tools/install.sh @@ -1824,6 +1824,10 @@ main() { return 0 fi + if [[ "${NO_PROMPT:-0}" != "1" ]] && [[ ! -t 0 ]] && { : /dev/null; then + exec Date: Mon, 25 May 2026 20:32:26 +0800 Subject: [PATCH 4/4] [docs] Remove outdated script description from installation instructions --- docs/content/docs/get-started/installation.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/content/docs/get-started/installation.md b/docs/content/docs/get-started/installation.md index 388810a43..c3e1afd7f 100644 --- a/docs/content/docs/get-started/installation.md +++ b/docs/content/docs/get-started/installation.md @@ -68,14 +68,6 @@ Run the one-liner from any directory you want the virtual environment created in curl -fsSL https://raw.githubusercontent.com/apache/flink-agents/main/tools/install.sh | bash ``` -### What the script does - -- Verifies that Java 11+ is available on your `PATH`. It does **not** install Java for you. -- Downloads and extracts Apache Flink into `$INSTALL_DIR` (default: `$HOME/.local/flink`), or reuses an existing `$FLINK_HOME` if you tell it not to install Flink. -- Creates a Python virtual environment (default: `./.flink-agents-env`) and runs `pip install flink-agents apache-flink` inside it. -- Copies `flink-python-.jar`, `flink-agents-dist-common-*.jar`, and the version-specific `flink-agents-dist-*` thin JAR into `$FLINK_HOME/lib`. -- Runs post-install verification (Flink binary, Python package, and JAR presence) before exiting. - ## Manual installation Use this path when you want full control over each step, when running `install.sh` is not feasible in your environment, or when building Flink Agents from source.