Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions fusion_docs/guide.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: User guide
description: "Overview of the Fusin v2 file system"
date: "23 Aug 2024"
description: "Overview of the Fusion v2 file system"
date created: "2024-08-23"
tags: [fusion, storage, compute, file system, posix, client]
---

Expand Down
11 changes: 7 additions & 4 deletions fusion_docs/guide/aws-batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: AWS Batch
description: "Use Fusion with AWS Batch and S3 storage"
date created: "2024-08-23"
last updated: "2025-09-09"
last updated: "2026-03-04"
tags: [fusion, storage, compute, aws-batch, s3]
---

Expand All @@ -13,7 +13,7 @@ Fusion simplifies and improves the efficiency of Nextflow pipelines in [AWS Batc
- Fusion uses an efficient data transfer and caching algorithm that provides much faster throughput compared to AWS CLI and does not require a local copy of data files.
- By replacing the AWS CLI with a native API client, the transfer is much more robust at scale.

### Platform AWS Batch compute environments
### Platform AWS Batch compute environments

Seqera Platform supports Fusion in Batch Forge and manual AWS Batch compute environments.

Expand Down Expand Up @@ -73,7 +73,7 @@ The scratch space at `/tmp` is shared by all tasks running on the instance. For

Configure with the following IAM permissions:

```
```json
{
"Version": "2012-10-17",
"Statement": [
Expand All @@ -89,9 +89,12 @@ Configure with the following IAM permissions:
{
"Action": [
"s3:GetObject",
"s3:GetObjectTagging",
"s3:PutObject",
"s3:PutObjectTagging",
"s3:DeleteObject"
"s3:DeleteObject",
"s3:CopyObject",
"s3:AbortMultipartUpload"
],
"Resource": [
"arn:aws:s3:::<S3_BUCKET>/*"
Expand Down
24 changes: 14 additions & 10 deletions fusion_docs/guide/aws-eks.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
---
title: Amazon EKS
description: "Use Fusion with AWS Elastic Kubernetes Service and S3 storage"
date: "23 Aug 2024"
date created: "2024-08-24"
last updated: "2026-03-04"
tags: [fusion, storage, compute, aws eks, s3]
---

Fusion streamlines the deployment of Nextflow pipelines in Kubernetes because it replaces the need to configure
and maintain a shared file system in your cluster.

### Platform Amazon EKS compute environments
### Platform Amazon EKS compute environments

Seqera Platform supports Fusion in Amazon Elastic Kubernetes Service (Amazon EKS) compute environments.
Seqera Platform supports Fusion in Amazon Elastic Kubernetes Service (Amazon EKS) compute environments.

See [Amazon EKS](https://docs.seqera.io/platform-cloud/compute-envs/eks) for Platform instructions to enable Fusion.

Expand All @@ -23,7 +24,7 @@ parallel to and from the object storage into the container-local temporary direc
Several AWS EC2 instance types include one or more NVMe SSD volumes. These volumes must be formatted to be used. See [SSD instance storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html) for details.
:::

To use Fusion directly in Nextflow with an Amazon EKS cluster, you must configure a namespace and service account and update your Nextflow configuration.
To use Fusion directly in Nextflow with an Amazon EKS cluster, you must configure a namespace and service account and update your Nextflow configuration.

#### Kubernetes configuration

Expand Down Expand Up @@ -84,12 +85,15 @@ Create a namespace and a service account in your Kubernetes cluster to run the j
"Resource": ["arn:aws:s3:::<S3_BUCKET>"]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectTagging",
"s3:DeleteObject"
],
"Action": [
"s3:GetObject",
"s3:GetObjectTagging",
"s3:PutObject",
"s3:PutObjectTagging",
"s3:DeleteObject",
"s3:CopyObject",
"s3:AbortMultipartUpload"
],
"Resource": ["arn:aws:s3:::<S3_BUCKET>/*"],
"Effect": "Allow"
}
Expand Down
8 changes: 4 additions & 4 deletions fusion_docs/guide/azure-batch.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Azure Batch
description: "Use Fusion with Azure Batch and Azure Blob storage"
date: "23 Aug 2024"
date created: "2024-08-23"
tags: [fusion, storage, compute, azure batch, blob storage]
---

Expand All @@ -12,9 +12,9 @@ Fusion simplifies and improves the efficiency of Nextflow pipelines in [Azure Ba
- By replacing the Azure CLI with a native API client, the transfer is much more robust at scale.
- By streaming relevant data and monitoring the virtual machine storage, Fusion can use more data than the capacity of the attached storage drive

### Platform Azure Batch compute environments
### Platform Azure Batch compute environments

Seqera Platform supports Fusion in Batch Forge and manual Azure Batch compute environments.
Seqera Platform supports Fusion in Batch Forge and manual Azure Batch compute environments.

See [Azure Batch](https://docs.seqera.io/platform-cloud/compute-envs/azure-batch) for compute and storage recommendations and instructions to enable Fusion.

Expand All @@ -30,7 +30,7 @@ We recommend selecting machine types with a local temp storage disk of at least
process.executor = 'azure-batch'
wave.enabled = true
fusion.enabled = true
tower.accessToken = '<PLATFORM_ACCESS_TOKEN>'
tower.accessToken = '<PLATFORM_ACCESS_TOKEN>'
```

Replace `<PLATFORM_ACCESS_TOKEN>` with your Platform access token.
Expand Down
8 changes: 4 additions & 4 deletions fusion_docs/guide/gcp-batch.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Google Cloud Batch
description: "Use Fusion with Google Cloud Batch and Google Cloud Storage"
date: "23 Aug 2024"
date created: "2024-08-23"
tags: [fusion, storage, compute, gcp batch, gcs, google cloud]
---

Expand All @@ -14,7 +14,7 @@ Fusion simplifies and improves the efficiency of Nextflow pipelines in [Google C

### Platform Google Cloud Batch compute environments

Seqera Platform supports Fusion in Google Cloud Batch compute environments.
Seqera Platform supports Fusion in Google Cloud Batch compute environments.

See [Google Cloud Batch](https://docs.seqera.io/platform-cloud/compute-envs/google-cloud-batch) for compute and storage recommendations and instructions to enable Fusion.

Expand All @@ -25,7 +25,7 @@ When Fusion v2 is enabled, the following virtual machine settings are applied:
- A 375 GB local NVMe SSD is selected for all compute jobs.
- If you do not specify a machine type, a VM from the following families that support local SSDs will be selected: `n1-*`, `n2-*`, `n2d-*`, `c2-*`, `c2d-*`, `m3-*`.
- Any machine types you specify in the Nextflow config must support local SSDs.
- Local SSDs are only offered in multiples of 375 GB. You can increment the number of SSDs used per process with the `disk` directive to request multiples of 375 GB.
- Local SSDs are only offered in multiples of 375 GB. You can increment the number of SSDs used per process with the `disk` directive to request multiples of 375 GB.
- Fusion v2 can also use persistent disks for caching. Override the disk requested by Fusion using the `disk` directive and the `type: pd-standard`.
- The `machineType` directive can be used to specify a VM instance type, family, or custom machine type in a comma-separated list of patterns. For example, `c2-*`, `n1-standard-1`, `custom-2-4`, `n*`, `m?-standard-*`.
:::
Expand All @@ -40,7 +40,7 @@ or with the `gcloud` auth application-default login command. See [Credentials](h
process.executor = 'google-batch'
wave.enabled = true
fusion.enabled = true
tower.accessToken = '<PLATFORM_ACCESS_TOKEN>'
tower.accessToken = '<PLATFORM_ACCESS_TOKEN>'
google.location = '<GOOGLE_LOCATION>'
```

Expand Down
10 changes: 5 additions & 5 deletions fusion_docs/guide/gcp-gke.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
---
title: Google Kubernetes Engine
description: "Use Fusion with Google Kubernetes Engine and Google Cloud Storage"
date: "23 Aug 2024"
date created: "2024-08-23"
tags: [fusion, storage, compute, gke, gcs, google cloud, kubernetes]
---

Fusion streamlines the deployment of Nextflow pipelines in Kubernetes because it replaces the need to configure
and maintain a shared file system in your cluster.

### Platform Google Kubernetes Engine compute environments
### Platform Google Kubernetes Engine compute environments

Seqera Platform supports Fusion in Google Kubernetes Engine (GKE) compute environments.
Seqera Platform supports Fusion in Google Kubernetes Engine (GKE) compute environments.

See [Google Kubernetes Engine](https://docs.seqera.io/platform-cloud/compute-envs/gke) for Platform instructions to enable Fusion.

Expand All @@ -20,7 +20,7 @@ See [Google Kubernetes Engine](https://docs.seqera.io/platform-cloud/compute-env
This feature requires Nextflow 23.02.1-edge or later.
:::

To use Fusion directly in Nextflow with a GKE cluster, you must configure a cluster, namespace, and service account, and update your Nextflow configuration.
To use Fusion directly in Nextflow with a GKE cluster, you must configure a cluster, namespace, and service account, and update your Nextflow configuration.

#### Kubernetes configuration

Expand Down Expand Up @@ -86,7 +86,7 @@ To use Fusion directly in Nextflow with a GKE cluster, you must configure a clus
process.scratch = false
wave.enabled = true
fusion.enabled = true
tower.accessToken = '<PLATFORM_ACCESS_TOKEN>'
tower.accessToken = '<PLATFORM_ACCESS_TOKEN>'
k8s.context = '<GKE_CLUSTER_CONTEXT>'
k8s.namespace = 'fusion-demo'
k8s.serviceAccount = 'fusion-sa'
Expand Down
39 changes: 38 additions & 1 deletion fusion_docs/guide/local/aws-s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ a large storage volume for temporary pipeline data.
This configuration requires Docker or a similar container engine to run pipeline tasks.
:::

1. Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to grant Nextflow and Fusion access to your storage credentials. See [Credentials](https://docs.seqera.io/nextflow/google#credentials) for more information.
1. Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to grant Nextflow and Fusion access to your storage credentials. See [Credentials](https://docs.seqera.io/nextflow/amazons3#security-credentials) for more information.

1. Add the following to your `nextflow.config` file:

Expand Down Expand Up @@ -47,3 +47,40 @@ To achieve optimal performance, set up an SSD volume as the temporary directory.
The option `fusion.exportStorageCredentials` leaks credentials on the task launcher script created by Nextflow.
This option should only be used for testing and development purposes.
:::


### IAM permissions

Configure with the following IAM permissions:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<S3_BUCKET>"
]
},
{
"Action": [
"s3:GetObject",
"s3:GetObjectTagging",
"s3:PutObject",
"s3:PutObjectTagging",
"s3:DeleteObject",
"s3:CopyObject",
"s3:AbortMultipartUpload"
],
"Resource": [
"arn:aws:s3:::<S3_BUCKET>/*"
],
"Effect": "Allow"
}
]
}
```
2 changes: 1 addition & 1 deletion fusion_docs/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Fusion file system
description: "Overview of the Fusion v2 file system"
date: "23 Aug 2024"
date created: "2024-08-23"
tags: [fusion, storage, compute, file system, posix, client]
---

Expand Down
6 changes: 3 additions & 3 deletions fusion_docs/licensing.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Fusion licensing
description: "Understand how Fusion licensing works"
date: "20 Feb 2025"
date created: "2025-02-20"
tags: [fusion, storage, compute, file system, license, quotas]
---

Expand All @@ -11,7 +11,7 @@ Fusion licensing requirements and free tier quotas will be enforced starting Jun

A Fusion license sets a specific quota for Fusion use, measured by total pipeline throughput per month (defined by [Nextflow’s I/O metrics](https://docs.seqera.io/nextflow/tutorials/metrics#i-o-usage)).

### For Seqera Cloud
### For Seqera Cloud

- Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform.
- By default, Seqera Cloud provides a monthly free tier for Fusion that allows up to 100 TB of throughput.
Expand All @@ -22,7 +22,7 @@ A Fusion license sets a specific quota for Fusion use, measured by total pipelin
- Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform.
- Once your quota is met or exceeded, you will be prevented from running additional Nextflow pipelines with Fusion.

### For Seqera Enterprise instances without network connectivity, and direct Fusion use with Nextflow
### For Seqera Enterprise instances without network connectivity, and direct Fusion use with Nextflow

- Run Fusion using the [`TOWER_ACCESS_TOKEN` environment variable](https://docs.seqera.io/nextflow/reference/config#tower) to allow Nextflow to connect to Seqera Cloud.
- Throughput counts towards the free or paid usage associated with your Cloud organization.
Loading