Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 97 additions & 86 deletions docs/content/docs/get-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,48 @@ The sections below show how to install the required dependencies.
__NOTE:__ To run on a Flink cluster, Flink-Agents requires a stable release of Flink version 1.20.3 or higher. We highly recommend using the **latest stable release of your chosen Flink minor version** (e.g., for minor version 2.2, use the latest 2.2.x release).
{{< /hint >}}

## Install Apache Flink
## Prerequisites

Both the script-based and manual installation paths require:

* **Java 11+** on your `PATH`. Java 21+ is recommended when using the Java API — see the table below for the version-specific limitations.
* **Python 3.10, 3.11, or 3.12** (only required if you plan to use the Python API or PyFlink).

For building Flink Agents from source, you additionally need:
- Unix-like environment (Linux, macOS, Cygwin, or WSL)
- Git
- Maven 3

### Java Versions

For running an agent built with the **Python API**, you can use any Java version 11 or higher.

When using the **Java API**, there are some functionality limitations for earlier Java versions, as detailed below:

| Java Version | Limitations |
|--------------|------------------------|
| Java 21+ | No limitations. |
| Java 11-20 | Async execution is unavailable. |

{{< hint info >}}
**Note**: If you want to use python 3.12, you need install Flink above 2.1 (including 2.1).
{{< /hint >}}

## Quick install (recommended)

The `install.sh` script provisions everything you need: it downloads Apache Flink, creates a Python virtual environment, installs `flink-agents` and `apache-flink` into it, and copies the required JARs into `$FLINK_HOME/lib`.

Run the one-liner from any directory you want the virtual environment created in:

```shell
curl -fsSL https://raw.githubusercontent.com/apache/flink-agents/main/tools/install.sh | bash
```

## Manual installation

Use this path when you want full control over each step, when running `install.sh` is not feasible in your environment, or when building Flink Agents from source.

### Install Apache Flink

Before installing Flink Agents, you need to have Apache Flink installed.

Expand All @@ -56,32 +97,6 @@ cp $FLINK_HOME/opt/flink-python-${FLINK_VERSION}.jar $FLINK_HOME/lib/
**Note:** For more detailed Flink installation instructions, refer to the [Flink local installation guide](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/try-flink/local_installation/).
{{< /hint >}}

{{< hint info >}}
**Note**: If you want to use python 3.12, you need install Flink above 2.1 (including 2.1).
{{< /hint >}}

## Install Flink Agents

### Prerequisites

* Python 3.10, 3.11 or 3.12
* For building from source, you also need:
- Unix-like environment (Linux, Mac OS X, Cygwin, or WSL)
- Git
- Maven 3
- Java 21+ (full functionality), or Java 11+ (some features unavailable)

### Java Versions

For running an agent built with **Python API**, you can use any Java version 11 or higher.

When using **Java API**, there are some functionality limitations for earlier Java versions, as detailed below:

| Java Version | Limitations |
|--------------|------------------------|
| Java 21+ | No limitations. |
| Java 11-20 | Async execution is unavailable. |

### Set Up Python Environment (Recommended)

We recommend using a Python virtual environment to isolate Flink Agents dependencies from your system Python packages.
Expand Down Expand Up @@ -158,6 +173,60 @@ After building:
- The Python package is installed and ready to use
- The distribution JAR is located at: `dist/flink-${FLINK_VERSION%.*}/target/flink-agents-dist-*.jar`

### Configure PYTHONPATH

Flink runs in its own JVM process and needs the `PYTHONPATH` environment variable to locate the flink-agents Python package. You need to set `PYTHONPATH` to the directory where flink-agents is installed.

**Determine your Python package installation path:**

The path depends on your Python environment setup:
- If using a virtual environment, it's the site-packages directory within your venv
- If using system Python, it's the system site-packages directory

**Tip:** You can use this command to help find the path:
```shell
python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
```

**Set PYTHONPATH before starting Flink:**

```shell
# Set PYTHONPATH to your Python site-packages directory

export PYTHONPATH=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')
```

{{< hint info >}}
**Note:** You can add the `export PYTHONPATH=...` line to your shell profile (`~/.bash_profile`, `~/.bashrc`, `~/.zprofile`, or `~/.zshrc`) to set it permanently. This way, it will be automatically configured in all future terminal sessions.
{{< /hint >}}

### Install Flink Agents Java Library

Copy the Flink Agents distribution JAR to your Flink installation's `lib` directory:

{{< tabs "Install Flink Agents Java Library" >}}

{{< tab "From Official Release" >}}
The Flink Agents JAR is bundled inside the Python package. Use the PYTHONPATH you configured above to locate and copy it:

```shell
# Copy the JAR from the Python package to Flink's lib directory
cp $PYTHONPATH/flink_agents/lib/flink-${FLINK_VERSION%.*}/flink-agents-dist-*.jar $FLINK_HOME/lib/
```

{{< /tab >}}

{{< tab "From Source" >}}
After building from source, the distribution JAR is located in the `dist/target/` directory:

```shell
# Copy the JAR to Flink's lib directory
cp dist/flink-${FLINK_VERSION%.*}/target/flink-agents-dist-*.jar $FLINK_HOME/lib/
```
{{< /tab >}}

{{< /tabs >}}

### Build Environment Variables

The following environment variables can be used to control how JARs are resolved during `pip install flink-agents` (from sdist) or `python -m build`:
Expand Down Expand Up @@ -229,67 +298,9 @@ For execution in IDE, enable the feature `add dependencies with provided scope t

## Deploy to Flink Cluster

After installing Flink Agents package, you need to deploy it to your Flink cluster so that Flink can run your agent jobs.

### Configure PYTHONPATH

Flink runs in its own JVM process and needs the `PYTHONPATH` environment variable to locate the flink-agents Python package. You need to set `PYTHONPATH` to the directory where flink-agents is installed.

**Determine your Python package installation path:**

The path depends on your Python environment setup:
- If using a virtual environment, it's the site-packages directory within your venv
- If using system Python, it's the system site-packages directory

**Tip:** You can use this command to help find the path:
```shell
python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'
```

**Set PYTHONPATH before starting Flink:**

```shell
# Set PYTHONPATH to your Python site-packages directory

export PYTHONPATH=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')
```

{{< hint info >}}
**Note:** You can add the `export PYTHONPATH=...` line to your shell profile (`~/.bash_profile`, `~/.bashrc`, `~/.zprofile`, or `~/.zshrc`) to set it permanently. This way, it will be automatically configured in all future terminal sessions.
{{< /hint >}}

### Install Flink Agents Java Library

Copy the Flink Agents distribution JAR to your Flink installation's `lib` directory:

{{< tabs "Install Flink Agents Java Library" >}}

{{< tab "From Official Release" >}}
The Flink Agents JAR is bundled inside the Python package. Use the PYTHONPATH you configured above to locate and copy it:

```shell
# Copy the JAR from the Python package to Flink's lib directory
cp $PYTHONPATH/flink_agents/lib/flink-${FLINK_VERSION%.*}/flink-agents-dist-*.jar $FLINK_HOME/lib/
```

{{< /tab >}}

{{< tab "From Source" >}}
After building from source, the distribution JAR is located in the `dist/target/` directory:

```shell
# Copy the JAR to Flink's lib directory
cp dist/flink-${FLINK_VERSION%.*}/target/flink-agents-dist-*.jar $FLINK_HOME/lib/
```
{{< /tab >}}

{{< /tabs >}}

### Start Flink Cluster or Submit Job

Once the Java library is installed and PYTHONPATH is configured, you can start your Flink cluster or submit jobs:

{{< tabs "Start Flink Cluster or Submit Job" >}}
{{< tabs "Deploy to Flink Cluster" >}}

{{< tab "Python" >}}
```bash
Expand Down
55 changes: 24 additions & 31 deletions tools/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -473,7 +473,6 @@ PYTHON_BIN="${PYTHON_BIN:-}"
NO_PROMPT="${NO_PROMPT:-0}"
VERBOSE="${FLINK_AGENTS_VERBOSE:-0}"
DRY_RUN="${FLINK_AGENTS_DRY_RUN:-0}"
VERIFY_INSTALL="${FLINK_AGENTS_VERIFY_INSTALL:-0}"
HELP=0
PYFLINK_ACTUALLY_ENABLED=0

Expand All @@ -491,7 +490,6 @@ Options:
--enable-pyflink Enable PyFlink and install Python packages
--verbose Print debug output (set -x)
--dry-run Print install plan without making changes
--verify Run post-install verification checks
--python <path> Path to a Python3 interpreter (overrides PATH lookup)
--flink-version <ver> Apache Flink version (e.g. 2.2.0); overrides the interactive picker
--flink-agents-version <ver>
Expand All @@ -513,7 +511,6 @@ Environment variables:
NO_PROMPT 1 to disable all prompts
FLINK_AGENTS_VERBOSE 1 to enable verbose output
FLINK_AGENTS_DRY_RUN 1 to enable dry-run mode
FLINK_AGENTS_VERIFY_INSTALL 1 to enable post-install verification

Examples:
bash install.sh --install-flink --enable-pyflink --non-interactive
Expand Down Expand Up @@ -1670,17 +1667,9 @@ show_install_plan() {
if [[ "$DRY_RUN" == "1" ]]; then
ui_kv "Dry run" "yes"
fi
if [[ "$VERIFY_INSTALL" == "1" ]]; then
ui_kv "Verify" "yes"
fi
}

verify_installation() {
if [[ "${VERIFY_INSTALL}" != "1" ]]; then
return 0
fi

ui_stage "Verifying installation"

if [[ -x "$FLINK_HOME/bin/flink" ]]; then
local flink_ver
Expand All @@ -1703,7 +1692,7 @@ verify_installation() {
fi

if [[ "$PYFLINK_ACTUALLY_ENABLED" -eq 1 ]]; then
if python -c "import flink_agents; print('flink-agents', flink_agents.__version__)" 2>/dev/null; then
if "$VENV_DIR/bin/python3" -c "import flink_agents; print('flink-agents ready')" 2>/dev/null; then
ui_success "flink-agents Python package verified"
else
ui_error "flink-agents Python package import failed"
Expand All @@ -1720,6 +1709,24 @@ verify_installation() {
ui_success "Verification complete"
}

show_next_step(){
echo ""
ui_celebrate "Apache Flink Agents installation finished!"
echo ""
ui_section "Next steps"
ui_success "1) Point FLINK_HOME at this install (Flink CLI and clients read it):"
ui_info " export FLINK_HOME=${FLINK_HOME}"
if [[ "$PYFLINK_ACTUALLY_ENABLED" -eq 1 ]]; then
ui_success "2) Activate the Python venv (PyFlink + flink-agents are installed there):"
ui_info " source ${VENV_DIR}/bin/activate"
ui_success "3) To make both permanent, append the two lines above to your shell rc"
ui_info " (~/.zshrc or ~/.bashrc)."
else
ui_success "2) To make it permanent, append the line above to your shell rc"
ui_info " (~/.zshrc or ~/.bashrc)."
fi
}

show_footer_links() {
local docs_url="https://nightlies.apache.org/flink/flink-agents-docs-latest/"
local issues_url="https://github.com/apache/flink-agents/issues"
Expand Down Expand Up @@ -1751,10 +1758,6 @@ parse_args() {
DRY_RUN=1
shift
;;
--verify)
VERIFY_INSTALL=1
shift
;;
--python)
if [[ $# -lt 2 ]]; then
die "--python requires a path argument"
Expand Down Expand Up @@ -1821,6 +1824,10 @@ main() {
return 0
fi

if [[ "${NO_PROMPT:-0}" != "1" ]] && [[ ! -t 0 ]] && { : </dev/tty; } 2>/dev/null; then
exec </dev/tty
fi

bootstrap_gum_temp || true
print_installer_banner
print_gum_status
Expand Down Expand Up @@ -1859,21 +1866,7 @@ main() {

verify_installation

echo ""
ui_celebrate "Apache Flink Agents installation finished!"
echo ""
ui_section "Next steps"
ui_success "1) Point FLINK_HOME at this install (Flink CLI and clients read it):"
ui_info " export FLINK_HOME=${FLINK_HOME}"
if [[ "$PYFLINK_ACTUALLY_ENABLED" -eq 1 ]]; then
ui_success "2) Activate the Python venv (PyFlink + flink-agents are installed there):"
ui_info " source ${VENV_DIR}/bin/activate"
ui_success "3) To make both permanent, append the two lines above to your shell rc"
ui_info " (~/.zshrc or ~/.bashrc)."
else
ui_success "2) To make it permanent, append the line above to your shell rc"
ui_info " (~/.zshrc or ~/.bashrc)."
fi
show_next_step

show_footer_links
}
Expand Down
1 change: 0 additions & 1 deletion tools/test/helpers/load.bash
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ reset_install_sh_state() {
DRY_RUN=0
NO_PROMPT=0
VERBOSE=0
VERIFY_INSTALL=0
INSTALL_FLINK="Ask"
ENABLE_PYFLINK="Ask"
PYTHON_BIN=""
Expand Down
8 changes: 1 addition & 7 deletions tools/test/unit/parse_args.bats
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,6 @@ setup() {
[ "$DRY_RUN" = "1" ]
}

@test "parse_args: --verify sets VERIFY_INSTALL=1" {
parse_args --verify
[ "$VERIFY_INSTALL" = "1" ]
}

@test "parse_args: --python <path> sets PYTHON_BIN" {
parse_args --python /opt/py/bin/python3
[ "$PYTHON_BIN" = "/opt/py/bin/python3" ]
Expand Down Expand Up @@ -74,9 +69,8 @@ setup() {
}

@test "parse_args: combined flags all apply" {
parse_args --non-interactive --install-flink --enable-pyflink --verify
parse_args --non-interactive --install-flink --enable-pyflink
[ "$NO_PROMPT" = "1" ]
[ "$INSTALL_FLINK" = "Yes" ]
[ "$ENABLE_PYFLINK" = "Yes" ]
[ "$VERIFY_INSTALL" = "1" ]
}
Loading