diff --git a/fusion_docs/guide.md b/fusion_docs/guide.md index dc5b0fb12..19f69f9b6 100644 --- a/fusion_docs/guide.md +++ b/fusion_docs/guide.md @@ -1,7 +1,7 @@ --- title: User guide -description: "Overview of the Fusin v2 file system" -date: "23 Aug 2024" +description: "Overview of the Fusion v2 file system" +date created: "2024-08-23" tags: [fusion, storage, compute, file system, posix, client] --- diff --git a/fusion_docs/guide/aws-batch.md b/fusion_docs/guide/aws-batch.md index 223fea3b3..a442bbf9e 100644 --- a/fusion_docs/guide/aws-batch.md +++ b/fusion_docs/guide/aws-batch.md @@ -2,7 +2,7 @@ title: AWS Batch description: "Use Fusion with AWS Batch and S3 storage" date created: "2024-08-23" -last updated: "2025-09-09" +last updated: "2026-03-04" tags: [fusion, storage, compute, aws-batch, s3] --- @@ -13,7 +13,7 @@ Fusion simplifies and improves the efficiency of Nextflow pipelines in [AWS Batc - Fusion uses an efficient data transfer and caching algorithm that provides much faster throughput compared to AWS CLI and does not require a local copy of data files. - By replacing the AWS CLI with a native API client, the transfer is much more robust at scale. -### Platform AWS Batch compute environments +### Platform AWS Batch compute environments Seqera Platform supports Fusion in Batch Forge and manual AWS Batch compute environments. @@ -73,7 +73,7 @@ The scratch space at `/tmp` is shared by all tasks running on the instance. For Configure with the following IAM permissions: -``` +```json { "Version": "2012-10-17", "Statement": [ @@ -89,9 +89,12 @@ Configure with the following IAM permissions: { "Action": [ "s3:GetObject", + "s3:GetObjectTagging", "s3:PutObject", "s3:PutObjectTagging", - "s3:DeleteObject" + "s3:DeleteObject", + "s3:CopyObject", + "s3:AbortMultipartUpload" ], "Resource": [ "arn:aws:s3:::/*" diff --git a/fusion_docs/guide/aws-eks.md b/fusion_docs/guide/aws-eks.md index 338e05130..39aced1cb 100644 --- a/fusion_docs/guide/aws-eks.md +++ b/fusion_docs/guide/aws-eks.md @@ -1,16 +1,17 @@ --- title: Amazon EKS description: "Use Fusion with AWS Elastic Kubernetes Service and S3 storage" -date: "23 Aug 2024" +date created: "2024-08-24" +last updated: "2026-03-04" tags: [fusion, storage, compute, aws eks, s3] --- Fusion streamlines the deployment of Nextflow pipelines in Kubernetes because it replaces the need to configure and maintain a shared file system in your cluster. -### Platform Amazon EKS compute environments +### Platform Amazon EKS compute environments -Seqera Platform supports Fusion in Amazon Elastic Kubernetes Service (Amazon EKS) compute environments. +Seqera Platform supports Fusion in Amazon Elastic Kubernetes Service (Amazon EKS) compute environments. See [Amazon EKS](https://docs.seqera.io/platform-cloud/compute-envs/eks) for Platform instructions to enable Fusion. @@ -23,7 +24,7 @@ parallel to and from the object storage into the container-local temporary direc Several AWS EC2 instance types include one or more NVMe SSD volumes. These volumes must be formatted to be used. See [SSD instance storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html) for details. ::: -To use Fusion directly in Nextflow with an Amazon EKS cluster, you must configure a namespace and service account and update your Nextflow configuration. +To use Fusion directly in Nextflow with an Amazon EKS cluster, you must configure a namespace and service account and update your Nextflow configuration. #### Kubernetes configuration @@ -84,12 +85,15 @@ Create a namespace and a service account in your Kubernetes cluster to run the j "Resource": ["arn:aws:s3:::"] }, { - "Action": [ - "s3:GetObject", - "s3:PutObject", - "s3:PutObjectTagging", - "s3:DeleteObject" - ], + "Action": [ + "s3:GetObject", + "s3:GetObjectTagging", + "s3:PutObject", + "s3:PutObjectTagging", + "s3:DeleteObject", + "s3:CopyObject", + "s3:AbortMultipartUpload" + ], "Resource": ["arn:aws:s3:::/*"], "Effect": "Allow" } diff --git a/fusion_docs/guide/azure-batch.md b/fusion_docs/guide/azure-batch.md index 26e69d72a..0a4d402f4 100644 --- a/fusion_docs/guide/azure-batch.md +++ b/fusion_docs/guide/azure-batch.md @@ -1,7 +1,7 @@ --- title: Azure Batch description: "Use Fusion with Azure Batch and Azure Blob storage" -date: "23 Aug 2024" +date created: "2024-08-23" tags: [fusion, storage, compute, azure batch, blob storage] --- @@ -12,9 +12,9 @@ Fusion simplifies and improves the efficiency of Nextflow pipelines in [Azure Ba - By replacing the Azure CLI with a native API client, the transfer is much more robust at scale. - By streaming relevant data and monitoring the virtual machine storage, Fusion can use more data than the capacity of the attached storage drive -### Platform Azure Batch compute environments +### Platform Azure Batch compute environments -Seqera Platform supports Fusion in Batch Forge and manual Azure Batch compute environments. +Seqera Platform supports Fusion in Batch Forge and manual Azure Batch compute environments. See [Azure Batch](https://docs.seqera.io/platform-cloud/compute-envs/azure-batch) for compute and storage recommendations and instructions to enable Fusion. @@ -30,7 +30,7 @@ We recommend selecting machine types with a local temp storage disk of at least process.executor = 'azure-batch' wave.enabled = true fusion.enabled = true - tower.accessToken = '' + tower.accessToken = '' ``` Replace `` with your Platform access token. diff --git a/fusion_docs/guide/gcp-batch.md b/fusion_docs/guide/gcp-batch.md index c02c7222c..15edc2691 100644 --- a/fusion_docs/guide/gcp-batch.md +++ b/fusion_docs/guide/gcp-batch.md @@ -1,7 +1,7 @@ --- title: Google Cloud Batch description: "Use Fusion with Google Cloud Batch and Google Cloud Storage" -date: "23 Aug 2024" +date created: "2024-08-23" tags: [fusion, storage, compute, gcp batch, gcs, google cloud] --- @@ -14,7 +14,7 @@ Fusion simplifies and improves the efficiency of Nextflow pipelines in [Google C ### Platform Google Cloud Batch compute environments -Seqera Platform supports Fusion in Google Cloud Batch compute environments. +Seqera Platform supports Fusion in Google Cloud Batch compute environments. See [Google Cloud Batch](https://docs.seqera.io/platform-cloud/compute-envs/google-cloud-batch) for compute and storage recommendations and instructions to enable Fusion. @@ -25,7 +25,7 @@ When Fusion v2 is enabled, the following virtual machine settings are applied: - A 375 GB local NVMe SSD is selected for all compute jobs. - If you do not specify a machine type, a VM from the following families that support local SSDs will be selected: `n1-*`, `n2-*`, `n2d-*`, `c2-*`, `c2d-*`, `m3-*`. - Any machine types you specify in the Nextflow config must support local SSDs. - - Local SSDs are only offered in multiples of 375 GB. You can increment the number of SSDs used per process with the `disk` directive to request multiples of 375 GB. + - Local SSDs are only offered in multiples of 375 GB. You can increment the number of SSDs used per process with the `disk` directive to request multiples of 375 GB. - Fusion v2 can also use persistent disks for caching. Override the disk requested by Fusion using the `disk` directive and the `type: pd-standard`. - The `machineType` directive can be used to specify a VM instance type, family, or custom machine type in a comma-separated list of patterns. For example, `c2-*`, `n1-standard-1`, `custom-2-4`, `n*`, `m?-standard-*`. ::: @@ -40,7 +40,7 @@ or with the `gcloud` auth application-default login command. See [Credentials](h process.executor = 'google-batch' wave.enabled = true fusion.enabled = true - tower.accessToken = '' + tower.accessToken = '' google.location = '' ``` diff --git a/fusion_docs/guide/gcp-gke.md b/fusion_docs/guide/gcp-gke.md index 5227ac1c6..49db8c689 100644 --- a/fusion_docs/guide/gcp-gke.md +++ b/fusion_docs/guide/gcp-gke.md @@ -1,16 +1,16 @@ --- title: Google Kubernetes Engine description: "Use Fusion with Google Kubernetes Engine and Google Cloud Storage" -date: "23 Aug 2024" +date created: "2024-08-23" tags: [fusion, storage, compute, gke, gcs, google cloud, kubernetes] --- Fusion streamlines the deployment of Nextflow pipelines in Kubernetes because it replaces the need to configure and maintain a shared file system in your cluster. -### Platform Google Kubernetes Engine compute environments +### Platform Google Kubernetes Engine compute environments -Seqera Platform supports Fusion in Google Kubernetes Engine (GKE) compute environments. +Seqera Platform supports Fusion in Google Kubernetes Engine (GKE) compute environments. See [Google Kubernetes Engine](https://docs.seqera.io/platform-cloud/compute-envs/gke) for Platform instructions to enable Fusion. @@ -20,7 +20,7 @@ See [Google Kubernetes Engine](https://docs.seqera.io/platform-cloud/compute-env This feature requires Nextflow 23.02.1-edge or later. ::: -To use Fusion directly in Nextflow with a GKE cluster, you must configure a cluster, namespace, and service account, and update your Nextflow configuration. +To use Fusion directly in Nextflow with a GKE cluster, you must configure a cluster, namespace, and service account, and update your Nextflow configuration. #### Kubernetes configuration @@ -86,7 +86,7 @@ To use Fusion directly in Nextflow with a GKE cluster, you must configure a clus process.scratch = false wave.enabled = true fusion.enabled = true - tower.accessToken = '' + tower.accessToken = '' k8s.context = '' k8s.namespace = 'fusion-demo' k8s.serviceAccount = 'fusion-sa' diff --git a/fusion_docs/guide/local/aws-s3.md b/fusion_docs/guide/local/aws-s3.md index 3fb5ddc3b..7ba506ae7 100644 --- a/fusion_docs/guide/local/aws-s3.md +++ b/fusion_docs/guide/local/aws-s3.md @@ -15,7 +15,7 @@ a large storage volume for temporary pipeline data. This configuration requires Docker or a similar container engine to run pipeline tasks. ::: -1. Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to grant Nextflow and Fusion access to your storage credentials. See [Credentials](https://docs.seqera.io/nextflow/google#credentials) for more information. +1. Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to grant Nextflow and Fusion access to your storage credentials. See [Credentials](https://docs.seqera.io/nextflow/amazons3#security-credentials) for more information. 1. Add the following to your `nextflow.config` file: @@ -47,3 +47,40 @@ To achieve optimal performance, set up an SSD volume as the temporary directory. The option `fusion.exportStorageCredentials` leaks credentials on the task launcher script created by Nextflow. This option should only be used for testing and development purposes. ::: + + +### IAM permissions + +Configure with the following IAM permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::" + ] + }, + { + "Action": [ + "s3:GetObject", + "s3:GetObjectTagging", + "s3:PutObject", + "s3:PutObjectTagging", + "s3:DeleteObject", + "s3:CopyObject", + "s3:AbortMultipartUpload" + ], + "Resource": [ + "arn:aws:s3:::/*" + ], + "Effect": "Allow" + } + ] +} +``` \ No newline at end of file diff --git a/fusion_docs/index.md b/fusion_docs/index.md index 8d4af341e..eaf91a828 100644 --- a/fusion_docs/index.md +++ b/fusion_docs/index.md @@ -1,7 +1,7 @@ --- title: Fusion file system description: "Overview of the Fusion v2 file system" -date: "23 Aug 2024" +date created: "2024-08-23" tags: [fusion, storage, compute, file system, posix, client] --- diff --git a/fusion_docs/licensing.md b/fusion_docs/licensing.md index 1e38ad407..bc81d68f4 100644 --- a/fusion_docs/licensing.md +++ b/fusion_docs/licensing.md @@ -1,7 +1,7 @@ --- title: Fusion licensing description: "Understand how Fusion licensing works" -date: "20 Feb 2025" +date created: "2025-02-20" tags: [fusion, storage, compute, file system, license, quotas] --- @@ -11,7 +11,7 @@ Fusion licensing requirements and free tier quotas will be enforced starting Jun A Fusion license sets a specific quota for Fusion use, measured by total pipeline throughput per month (defined by [Nextflow’s I/O metrics](https://docs.seqera.io/nextflow/tutorials/metrics#i-o-usage)). -### For Seqera Cloud +### For Seqera Cloud - Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform. - By default, Seqera Cloud provides a monthly free tier for Fusion that allows up to 100 TB of throughput. @@ -22,7 +22,7 @@ A Fusion license sets a specific quota for Fusion use, measured by total pipelin - Seqera Platform will automatically add and manage a Fusion license on your behalf, up to the allowed usage quota. Seqera will track Fusion use for all pipelines run in Platform. - Once your quota is met or exceeded, you will be prevented from running additional Nextflow pipelines with Fusion. -### For Seqera Enterprise instances without network connectivity, and direct Fusion use with Nextflow +### For Seqera Enterprise instances without network connectivity, and direct Fusion use with Nextflow - Run Fusion using the [`TOWER_ACCESS_TOKEN` environment variable](https://docs.seqera.io/nextflow/reference/config#tower) to allow Nextflow to connect to Seqera Cloud. - Throughput counts towards the free or paid usage associated with your Cloud organization.