-
diff --git a/docs/calypr/data/managing-metadata.md b/docs/calypr/data/managing-metadata.md
new file mode 100644
index 0000000..e6b85b6
--- /dev/null
+++ b/docs/calypr/data/managing-metadata.md
@@ -0,0 +1,182 @@
+# Managing Metadata
+
+Metadata in Calypr is formatted using the Fast Healthcare Interoperability Resources (FHIR) schema. If you choose to bring your own FHIR newline delimited json data, you will need to create a directory called “META” in your git-drs repository in the same directory that you initialized your git-drs repository, and place your metadata files in that directory.
+The META/ folder contains newline-delimited JSON (.ndjson) files representing FHIR resources describing the project, its data, and related entities. Large files are tracked using Git LFS, with a required correlation between each data file and a DocumentReference resource. This project follows a standardized structure to manage large research data files and associated FHIR metadata in a version-controlled, DRS and FHIR compatible format.
+Each file must contain only one type of FHIR resource type, for example META/ResearchStudy.ndjson only contains research study resource typed FHIR objects. The name of the file doesn’t have to match the resource type name, unless you bring your own document references, then you must use DocumentReference.ndjson. For all other FHIR file types this is simply a good organizational practice for organizing your FHIR metadata.
+
+## META/ResearchStudy.ndjson
+
+* The File directory structure root research study is based on the 1st Research Study in the document. This research study is the research study that the autogenerated document references are connected to. Any additional research studies that are provided will be ignored when populating the miller table file tree.
+* Contains at least one FHIR ResearchStudy resource describing the project.
+* Defines project identifiers, title, description, and key attributes.
+
+## META/DocumentReference.ndjson
+
+* Contains one FHIR DocumentReference resource per Git LFS-managed file.
+* Each `DocumentReference.content.attachment.url` field:
+ * Must exactly match the relative path of the corresponding file in the repository.
+ * Example:
+
+```json
+{
+ "resourceType": "DocumentReference",
+ "id": "docref-file1",
+ "status": "current",
+ "content": [
+ {
+ "attachment": {
+ "url": "data/file1.bam",
+ "title": "BAM file for Sample X"
+ }
+ }
+ ]
+}
+```
+
+Place your custom FHIR `.ndjson` files in the `META/` directory:
+
+```bash
+# Copy your prepared FHIR metadata
+cp ~/my-data/patients.ndjson META/
+cp ~/my-data/observations.ndjson META/
+cp ~/my-data/specimens.ndjson META/
+cp ~/my-data/document-references.ndjson META/
+```
+
+## Other FHIR data
+
+\[TODO More intro text here\]
+
+* Patient.ndjson: Participant records.
+* Specimen.ndjson: Biological specimens.
+* ServiceRequest.ndjson: Requested procedures.
+* Observation.ndjson: Measurements or results.
+* Other valid FHIR resource types as required.
+
+Ensure your FHIR `DocumentReference` resources reference the DRS URIs:
+
+Example `DocumentReference` linking to S3 file:
+
+```json
+{
+ "resourceType": "DocumentReference",
+ "id": "doc-001",
+ "status": "current",
+ "content": [{
+ "attachment": {
+ "url": "drs://calypr-public.ohsu.edu/your-drs-id",
+ "title": "sample1.bam",
+ "contentType": "application/octet-stream"
+ }
+ }],
+ "subject": {
+ "reference": "Patient/patient-001"
+ }
+}
+```
+
+
+---
+
+## Validating Metadata
+
+To ensure that the FHIR files you have added to the project are correct and pass schema checking, you can use the [Forge tool](../../tools/forge/index.md).
+
+```bash
+forge validate
+```
+
+Successful output:
+
+✓ Validating META/patients.ndjson... OK
+✓ Validating META/observations.ndjson... OK
+✓ Validating META/specimens.ndjson... OK
+✓ Validating META/document-references.ndjson... OK
+All metadata files are valid.
+
+Fix any validation errors and re-run until all files pass.
+
+
+### Forge Data Quality Assurance Command Line Commands
+
+If you have provided your own FHIR resources there are two commands that might be useful to you for ensuring that your FHIR metadata will appear on the CALYPR data platform as expected. These commands are validate and check-edge
+
+**Validate:**
+```bash
+forge validate META
+# or
+forge validate META/DocumentReference.ndjson
+```
+Validation checks if the provided directory or file will be accepted by the CALYPR data platform. It catches improper JSON formatting and FHIR schema errors.
+
+**Check-edge:**
+```bash
+forge check-edge META
+# or
+forge validate META/DocumentReference.ndjson
+```
+Check-edge ensures that references within your files (e.g., a Patient ID in an Observation) connect to known vertices and aren't "orphaned".
+
+### Validation Process
+
+#### 1\. Schema Validation
+
+* Each .ndjson file in META/ (like ResearchStudy.ndjson, DocumentReference.ndjson, etc.) is read line by line.
+* Every line is parsed as JSON and checked against the corresponding FHIR schema for that resourceType.
+* Syntax errors, missing required fields, or invalid FHIR values trigger clear error messages with line numbers.
+
+#### 2\. Mandatory Files Presence
+
+* Confirms that:
+ * ResearchStudy.ndjson exists and has at least one valid record.
+ * DocumentReference.ndjson exists and contains at least one record.
+* If either is missing or empty, validation fails.
+
+#### 3\. One-to-One Mapping of Files to DocumentReference
+
+* Scans the working directory for Git LFS-managed files in expected locations (e.g., data/).
+* For each file, locates a corresponding DocumentReference resource whose content.attachment.url matches the file’s relative path.
+* Validates:
+ * All LFS files have a matching DocumentReference.
+ * All DocumentReferences point to existing files.
+
+#### 4\. Project-level Referential Checks
+
+* Validates that DocumentReference resources reference the same ResearchStudy via relatesTo or other linking mechanisms.
+* If FHIR resources like Patient, Specimen, ServiceRequest, Observation are present, ensures:
+ * Their id fields are unique.
+ * DocumentReference correctly refers to those resources (e.g., via subject or related fields).
+
+#### 5\. Cross-Entity Consistency
+
+* If multiple optional FHIR .ndjson files exist:
+ * Confirms IDs referenced in one file exist in others.
+ * Detects dangling references (e.g., a DocumentReference.patient ID that's not in Patient.ndjson).
+
+---
+
+#### ✅ Example Error Output
+
+ERROR META/DocumentReference.ndjson line 4: url "data/some\_missing.bam" does not resolve to an existing file
+ERROR META/Specimen.ndjson line 2: id "specimen-123" referenced in Observation.ndjson but not defined
+
+---
+
+#### 🎯 Purpose & Benefits
+
+* Ensures all files and metadata are in sync before submission.
+* Prevents submission failures due to missing pointers or invalid FHIR payloads.
+* Enables CI integration, catching issues early in the development workflow.
+
+---
+
+#### Validation Requirements
+
+Automated tools or CI processes must:
+
+* Verify presence of META/ResearchStudy.ndjson with at least one record.
+* Verify presence of META/DocumentReference.ndjson with one record per LFS-managed file.
+* Confirm every DocumentReference.url matches an existing file path.
+* Check proper .ndjson formatting.
+
+---
\ No newline at end of file
diff --git a/docs/calypr/index.md b/docs/calypr/index.md
new file mode 100644
index 0000000..d0ee9f2
--- /dev/null
+++ b/docs/calypr/index.md
@@ -0,0 +1,43 @@
+# CALYPR Platform
+
+Welcome to the **CALYPR Platform**. CALYPR is a next-generation genomic data science ecosystem designed to bridge the gap between massive, centralized data commons and the agile, distributed workflows of modern researchers.
+
+
+## The CALYPR Philosophy
+
+Traditional data repositories often create data silos where information is easy to store but difficult to move, version, or integrate with external tools. CALYPR breaks these silos by embracing **Interoperability**, **Reproducibility**, and **Scalability**.
+
+### 1. Interoperability (GA4GH Standards)
+CALYPR is built from the ground up on [GA4GH](https://www.ga4gh.org/) standards. By using the **Data Repository Service (DRS)** and **Task Execution Service (TES)**, CALYPR ensures that your data and workflows can move seamlessly between different cloud providers and on-premises high-performance computing (HPC) clusters.
+The data system is based on the Fast Healthcare Interoperability Resources (FHIR) standard.
+
+### 2. Reproducibility (Git-like Data Management)
+The core of the CALYPR experience is **Git-DRS**. We believe that data management should feel as natural as code management. Git-DRS allows you to track, version, and share massive genomic datasets using the familiar `git` commands, ensuring that every analysis is backed by a specific, immutable version of the data.
+
+### 3. Scalability (Hybrid Cloud Infrastructure)
+Whether you are working with a few genomes or petabyte-scale cohorts, CALYPR's architecture—powered by **Gen3**—scales to meet your needs. Our hybrid cloud approach allows for secure data storage in AWS while leveraging your local compute resources when necessary.
+
+---
+
+## How it Works: The Connected Commons
+
+CALYPR acts as the "connective tissue" between your research environment and the cloud:
+
+* **Data Commons ([Gen3](https://gen3.org)):** Provides the robust backend for metadata management, indexing, and authentication.
+* **Version Control ([Git-DRS](../tools/git-drs/index.md)):** Manages the check-in and check-out operations for large files, allowing you to treat remote DRS objects as local files.
+* **Metadata Orchestration ([Forge](../tools/forge/index.md)):** Streamlines the validation, publishing, and harmonizing of genomic metadata.
+* **Compute ([Funnel](../tools/funnel/index.md)):** Executes complex pipelines across distributed environments using standardized task definitions.
+* **Graph Insights ([GRIP](../tools/grip/index.md)):** Enables high-performance queries across heterogeneous datasets once integrated.
+
+---
+
+!!! info "Private Beta"
+ CALYPR platform is currently in a private beta phase. We are actively working with a select group of research partners to refine the platform. If you encounter any issues or have feature requests, please reach out to the team. The individual [tools](../tools/index.md) are available for public use.
+
+---
+
+## Next Steps
+To get started:
+
+1. **[Quick Start Guide](quick-start.md):** The fastest way to install tools and start tracking data.
+2. **[Data & Metadata](data/managing-metadata.md):** Learn how to associate your biological metadata with the files you've uploaded.
\ No newline at end of file
diff --git a/docs/calypr/project-management/.nav.yml b/docs/calypr/project-management/.nav.yml
new file mode 100644
index 0000000..ce24a4e
--- /dev/null
+++ b/docs/calypr/project-management/.nav.yml
@@ -0,0 +1,4 @@
+nav:
+ - Project Customization: custom-views.md
+ - Publishing project: publishing-project.md
+ - Calypr Admin: calypr-admin/
diff --git a/docs/calypr/project-management/calypr-admin/.nav.yml b/docs/calypr/project-management/calypr-admin/.nav.yml
new file mode 100644
index 0000000..bfa1166
--- /dev/null
+++ b/docs/calypr/project-management/calypr-admin/.nav.yml
@@ -0,0 +1,3 @@
+nav:
+ - Add users: add-users.md
+ - Role Based Access Control: approve-requests.md
diff --git a/docs/workflows/add-users.md b/docs/calypr/project-management/calypr-admin/add-users.md
similarity index 72%
rename from docs/workflows/add-users.md
rename to docs/calypr/project-management/calypr-admin/add-users.md
index 86cedac..8589726 100644
--- a/docs/workflows/add-users.md
+++ b/docs/calypr/project-management/calypr-admin/add-users.md
@@ -3,7 +3,7 @@
## Granting user access to a project
Once a project has been created you will have full access to it.
-The project owner can add additional users to the project using the `g3t collaborator` commands.
+The project owner can add additional users to the project using the `data-client collaborator` commands.
There are two ways to request the addition additional users to the project:
@@ -12,15 +12,14 @@ There are two ways to request the addition additional users to the project:
To give another user full access to the project, run the following:
```sh
-g3t collaborator add --write user-can-write@example.com
+data-client collaborator add [project_id] user-can-write@example.com --write
```
Alternatively, to give another user read access only (without the ability to upload to the project), run the following:
```sh
-g3t collaborator add user-read-only@example.com
+data-client collaborator add [project_id] user-read-only@example.com
```
## 2. Approvals
-In order to implement these requests, **an authorized user will need to sign** the request before the user can use the remote repository. See `g3t collaborator approve --help
-`
+In order to implement these requests, **an authorized user will need to sign** the request before the user can use the remote repository. See `data-client collaborator approve --help`
diff --git a/docs/workflows/approve-requests.md b/docs/calypr/project-management/calypr-admin/approve-requests.md
similarity index 90%
rename from docs/workflows/approve-requests.md
rename to docs/calypr/project-management/calypr-admin/approve-requests.md
index 0c836d0..2577139 100644
--- a/docs/workflows/approve-requests.md
+++ b/docs/calypr/project-management/calypr-admin/approve-requests.md
@@ -16,8 +16,8 @@
* Ony users with the steward role can approve and sign a request
```text
-g3t collaborator approve --help
-Usage: g3t collaborator approve [OPTIONS]
+./data-client collaborator approve --help
+Usage: ./data-client collaborator approve [OPTIONS]
Sign an existing request (privileged).
@@ -40,9 +40,9 @@ Note: This example uses the ohsu program, but the same process applies to all pr
```text
## As an admin, I need to grant data steward privileges add the requester reader and updater role on a program to an un-privileged user
-g3t collaborator add add data_steward_example@
+
-## About
-The [gen3-tracker](https://github.com/CALYPR/gen3_util/) (g3t) command line utility is a combination of tools that facilitate data sharing on the CALYPR platform. It allows you to create a unified data project, upload files, and associate those files with metadata in an incremental manner. Submitted data with g3t gives you all the benefits the data platform offers: data indexing, data exploration, consolidated access, and more!
+CALYPR Platform
++ A scalable, hybrid cloud infrastructure designed for the demands of modern genomics research. + Built on open-source standards, CALYPR provides GA4GH-compliant tools for seamless data integration, analysis, and biological insights. Based on the Gen3 Data Commons architecture, CALYPR empowers analysts to manage large-scale genomic datasets and integrate data to build new predictive models. +
+ +-The following guide details the steps a data contributor must take to submit a project to the CALYPR data commons. -## Getting Started +
+
-To navigate through each page, use pages list in the top left or using the navigation arrow on the bottom left and right! Otherwise, check out our [requirements](requirements.md) page to get started.
+Built on Open Standards
+
+
+
+
+
+ TES
+ Task Execution Service. A GA4GH standard for distributed task execution to enable federated computing.
+
+
+ DRS
+ Data Reference System. A GA4GH standard for data discovery and access.
+
+
+ FHIR
+ Healthcare Interoperability. Exchanging patient health information.
+
+
+ JSON Hyper-Schema
+ JSON-Schema + Graph data. Represent complex and high quality data.
+ - +
+
+
+
+
+
+
+
+
+ 
+
+
+
+
+ Scalable genomics data science platform for biological insights.
+Next-generation genomics data science platform with scalable cloud / on-prem hybrid infrastructure, streamlining the journey from raw data to discovery.
+ Learn more → +
+
+
+
+
+ 
+
+
+
+
+ GRIP
+Graph-based data integration for complex research datasets.
+High-performance graph query engine that provides a unified interface across MongoDB, SQL, and key-value stores. Ideal for complex relational discovery in genomics.
+ Learn more → +
+
+
+
+
+ 
+
+
+
+
+ Funnel
+Distributed task execution for petabyte-scale pipelines.
+Standardized batch computing using the GA4GH TES API. Run Docker-based tasks seamlessly across AWS, Google Cloud, and Kubernetes at any scale.
+ Learn more → +
+
+
+ 
+
+
+
+
+ Git-DRS
+Secure data repository system with version control.
+Manage large-scale genomic data with integrated versioning and metadata management, ensuring reproducibility and data integrity throughout research cycles.
+ Learn more → ++ +
+
diff --git a/docs/note.md b/docs/note.md
deleted file mode 100644
index 1b65636..0000000
--- a/docs/note.md
+++ /dev/null
@@ -1,2 +0,0 @@
-!!! note
- The tools listed here are under development and may be subject to change.
diff --git a/docs/requirements.md b/docs/requirements.md
deleted file mode 100644
index 62eb900..0000000
--- a/docs/requirements.md
+++ /dev/null
@@ -1,154 +0,0 @@
----
-title: Requirements
----
-
-# Requirements
-
-## 1. Download gen3-client
-
-gen3-client to upload and download files to the [gen3 platform](https://gen3.org/). Since the CALYPR is built on gen3, gen3-client is used in gen3-tracker (g3t) for the same purpose. See the instructions below for how to download gen3-client for your operating system.
-
-### Installation Instructions
-
-
-=== "macOS"
- 1. Download the [macOS version](https://github.com/CALYPR/cdis-data-client/releases/latest/download/gen3-client-macos.pkg) of the gen3-client.
- 2. Run the gen3-client pkg, following the instructions in the installer.
- 3. Open a terminal window.
- 4. Create a new gen3 directory: `mkdir ~/.gen3`
- 5. Move the executable to the gen3 directory: `mv /Applications/gen3-client ~/.gen3/gen3-client`
- 6. Change file permissions: `chown $USER ~/.bash_profile`
- 7. Add the gen3 directory to your PATH environment variable: `echo 'export PATH=$PATH:~/.gen3' >> ~/.bash_profile`
- 8. Refresh your PATH: `source ~/.bash_profile`
- 9. Check that the program is downloaded: run `gen3-client`
-
-
-=== "Linux"
- 1. Download the [Linux version](https://github.com/CALYPR/cdis-data-client/releases/latest/download/gen3-client-linux-amd64.zip) of the gen3-client.
- 2. Unzip the archive.
- 3. Open a terminal window.
- 4. Create a new gen3 directory: `mkdir ~/.gen3`
- 5. Move the unzipped executable to the gen3 directory: `~/.gen3/gen3-client`
- 6. Change file permissions: `chown $USER ~/.bash_profile`
- 7. Add the gen3 directory to your PATH environment variable: `echo 'export PATH=$PATH:~/.gen3' >> ~/.bash_profile`
- 8. Refresh your PATH: `source ~/.bash_profile`
- 9. Check that the program is downloaded: run `gen3-client`
-
-=== "Windows"
- 1. Download the [Windows version](https://github.com/CALYPR/cdis-data-client/releases/latest/download/gen3-client-windows-amd64.zip) of the gen3-client.
- 2. Unzip the archive.
- 3. Add the unzipped executable to a directory, for example: `C:\Program Files\gen3-client\gen3-client.exe`
- 4. Open the Start Menu and type "edit environment variables".
- 5. Open the option "Edit the system environment variables".
- 6. In the "System Properties" window that opens up, on the "Advanced" tab, click on the "Environment Variables" button.
- 7. In the box labeled "System Variables", find the "Path" variable and click "Edit".
- 8. In the window that pops up, click "New".
- 9. Type in the full directory path of the executable file, for example: `C:\Program Files\gen3-client`
- 10. Click "Ok" on all the open windows and restart the command prompt if it is already open by entering cmd into the start menu and hitting enter.
-
-## 2. Configure a gen3-client Profile with Credentials
-
-To use the gen3-client, you need to configure `gen3-client` with API credentials downloaded from the [Profile page](https://calypr.ohsu.edu.org/Profile).
-
-
-
-Log into the website. Then, download the access key from the portal and save it in the standard location `~/.gen3/credentials.json`
-
-
-
-From the command line, run the gen3-client configure command:
-
-=== "Example Command"
- ```sh
- gen3-client configure \
- --profile=Join the Beta
++ CALYPR is currently in private beta. If you are interested in early access or a demonstration of the platform, please reach out to us at + support@calypr.org. In the meantime, you can explore our GitHub repository and get access to all of our open source tools. +
+
+
+
+# Additional Resources 📚
+
+- [Helm Repo](https://ohsu-comp-bio.github.io/helm-charts)
+
+- [Helm Repo Source](https://github.com/ohsu-comp-bio/helm-charts)
+
+- [Helm Charts](https://github.com/ohsu-comp-bio/funnel/tree/develop/deployments/kubernetes/helm)
+
+- [The Chart Best Practices Guide](https://helm.sh/docs/chart_best_practices/)
diff --git a/docs/tools/funnel/docs/compute/pbs-torque.md b/docs/tools/funnel/docs/compute/pbs-torque.md
new file mode 100644
index 0000000..59b2dfa
--- /dev/null
+++ b/docs/tools/funnel/docs/compute/pbs-torque.md
@@ -0,0 +1,57 @@
+---
+title: PBS/Torque
+render_macros: true
+menu:
+ main:
+ parent: Compute
+ weight: 20
+---
+# PBS/Torque
+
+Funnel can be configured to submit workers to [PBS/Torque][pbs] by making calls
+to `qsub`.
+
+The Funnel server needs to run on a submission node.
+Configure Funnel to use PBS by including the following config:
+
+It is recommended to update the submit file template so that the
+`funnel worker run` command takes a config file as an argument
+(e.g. `funnel worker run --config /opt/funnel_config.yml --taskID {% raw %}{{.TaskId}}{% endraw %}`)
+
+{% raw %}
+```YAML
+Compute: pbs
+
+PBS:
+ Template: |
+ #!/bin/bash
+ #PBS -N {{.TaskId}}
+ #PBS -o {{.WorkDir}}/funnel-stdout
+ #PBS -e {{.WorkDir}}/funnel-stderr
+ {{if ne .Cpus 0 -}}
+ {{printf "#PBS -l nodes=1:ppn=%d" .Cpus}}
+ {{- end}}
+ {{if ne .RamGb 0.0 -}}
+ {{printf "#PBS -l mem=%.0fgb" .RamGb}}
+ {{- end}}
+ {{if ne .DiskGb 0.0 -}}
+ {{printf "#PBS -l file=%.0fgb" .DiskGb}}
+ {{- end}}
+
+ funnel worker run --taskID {{.TaskId}}
+```
+{% endraw %}
+The following variables are available for use in the template:
+
+| Variable | Description |
+|:------------|:-------------|
+|TaskId | funnel task id |
+|WorkDir | funnel working directory |
+|Cpus | requested cpu cores |
+|RamGb | requested ram |
+|DiskGb | requested free disk space |
+|Zone | requested zone (could be used for queue name) |
+
+See https://golang.org/pkg/text/template for information on creating templates.
+
+[pbs]: https://hpc-wiki.info/hpc/Torque
diff --git a/docs/tools/funnel/docs/compute/slurm.md b/docs/tools/funnel/docs/compute/slurm.md
new file mode 100644
index 0000000..98f1697
--- /dev/null
+++ b/docs/tools/funnel/docs/compute/slurm.md
@@ -0,0 +1,57 @@
+---
+title: Slurm
+menu:
+ main:
+ parent: Compute
+ weight: 20
+---
+# Slurm
+
+Funnel can be configured to submit workers to [Slurm][slurm] by making calls
+to `sbatch`.
+
+The Funnel server needs to run on a submission node.
+Configure Funnel to use Slurm by including the following config:
+
+It is recommended to update the submit file template so that the
+`funnel worker run` command takes a config file as an argument
+(e.g. `funnel worker run --config /opt/funnel_config.yml --taskID {% raw %}{{.TaskId}}{% endraw %}`)
+
+{% raw %}
+```YAML
+Compute: slurm
+
+Slurm:
+ Template: |
+ #!/bin/bash
+ #SBATCH --job-name {{.TaskId}}
+ #SBATCH --ntasks 1
+ #SBATCH --error {{.WorkDir}}/funnel-stderr
+ #SBATCH --output {{.WorkDir}}/funnel-stdout
+ {{if ne .Cpus 0 -}}
+ {{printf "#SBATCH --cpus-per-task %d" .Cpus}}
+ {{- end}}
+ {{if ne .RamGb 0.0 -}}
+ {{printf "#SBATCH --mem %.0fGB" .RamGb}}
+ {{- end}}
+ {{if ne .DiskGb 0.0 -}}
+ {{printf "#SBATCH --tmp %.0fGB" .DiskGb}}
+ {{- end}}
+
+ funnel worker run --taskID {{.TaskId}}
+```
+{% endraw %}
+The following variables are available for use in the template:
+
+| Variable | Description |
+|:------------|:-------------|
+|TaskId | funnel task id |
+|WorkDir | funnel working directory |
+|Cpus | requested cpu cores |
+|RamGb | requested ram |
+|DiskGb | requested free disk space |
+|Zone | requested zone (could be used for queue name) |
+
+See https://golang.org/pkg/text/template for information on creating templates.
+
+[slurm]: https://slurm.schedmd.com/
diff --git a/docs/tools/funnel/docs/databases.md b/docs/tools/funnel/docs/databases.md
new file mode 100644
index 0000000..5eeb638
--- /dev/null
+++ b/docs/tools/funnel/docs/databases.md
@@ -0,0 +1,8 @@
+---
+title: Databases
+menu:
+ main:
+ weight: 5
+---
+
+# Databases
diff --git a/docs/tools/funnel/docs/databases/boltdb.md b/docs/tools/funnel/docs/databases/boltdb.md
new file mode 100644
index 0000000..ea5885e
--- /dev/null
+++ b/docs/tools/funnel/docs/databases/boltdb.md
@@ -0,0 +1,24 @@
+---
+title: Embedded
+menu:
+ main:
+ parent: Databases
+ weight: -10
+---
+
+# Embedded
+
+By default, Funnel uses an embedded database named [BoltDB][bolt] to store task
+and scheduler data. This is great for development and a simple server without
+external dependencies, but it doesn't scale well to larger clusters.
+
+Available config:
+```yaml
+Database: boltdb
+
+BoltDB:
+ # Path to database file
+ Path: ./funnel-work-dir/funnel.db
+```
+
+[bolt]: https://github.com/boltdb/bolt
diff --git a/docs/tools/funnel/docs/databases/datastore.md b/docs/tools/funnel/docs/databases/datastore.md
new file mode 100644
index 0000000..ea31d8c
--- /dev/null
+++ b/docs/tools/funnel/docs/databases/datastore.md
@@ -0,0 +1,94 @@
+---
+title: Datastore
+menu:
+ main:
+ parent: Databases
+---
+
+# Google Cloud Datastore
+
+Funnel supports storing tasks (but not scheduler data) in Google Cloud Datastore.
+
+This implementation currently doesn't work with Appengine, since Appengine places
+special requirements on the context of requests and requires a separate library.
+
+Two entity types are used, "Task" and "TaskPart" (for larger pieces of task content,
+such as stdout/err logs).
+
+Funnel will, by default, try to automatically load credentials from the
+environment. Alternatively, you may explicitly set the credentials in the config.
+You can read more about providing the credentials
+[here](https://cloud.google.com/docs/authentication/application-default-credentials).
+
+Config:
+```yaml
+Database: datastore
+
+Datastore:
+ Project: ""
+ # Path to account credentials file.
+ # Optional. If possible, credentials will be automatically discovered
+ # from the environment.
+ CredentialsFile: ""
+```
+
+Please also import some [composite
+indexes](https://cloud.google.com/datastore/docs/concepts/indexes?hl=en)
+to support the task-list queries.
+This is typically done through command-line by referencing an **index.yaml**
+file (do not change the filename) with the following content:
+
+```shell
+gcloud datastore indexes create path/to/index.yaml --database='funnel'
+```
+
+```yaml
+indexes:
+
+- kind: Task
+ properties:
+ - name: Owner
+ - name: State
+ - name: TagStrings
+ - name: CreationTime
+ direction: desc
+
+- kind: Task
+ properties:
+ - name: Owner
+ - name: State
+ - name: CreationTime
+ direction: desc
+
+- kind: Task
+ properties:
+ - name: Owner
+ - name: TagStrings
+ - name: CreationTime
+ direction: desc
+
+- kind: Task
+ properties:
+ - name: Owner
+ - name: CreationTime
+ direction: desc
+
+- kind: Task
+ properties:
+ - name: State
+ - name: TagStrings
+ - name: CreationTime
+ direction: desc
+
+- kind: Task
+ properties:
+ - name: State
+ - name: CreationTime
+ direction: desc
+
+- kind: Task
+ properties:
+ - name: TagStrings
+ - name: CreationTime
+ direction: desc
+```
\ No newline at end of file
diff --git a/docs/tools/funnel/docs/databases/dynamodb.md b/docs/tools/funnel/docs/databases/dynamodb.md
new file mode 100644
index 0000000..3e536c2
--- /dev/null
+++ b/docs/tools/funnel/docs/databases/dynamodb.md
@@ -0,0 +1,30 @@
+---
+title: DynamoDB
+menu:
+ main:
+ parent: Databases
+---
+
+# DynamoDB
+
+Funnel supports storing task data in DynamoDB. Storing scheduler data is not supported currently, so using the node scheduler with DynamoDB won't work. Using AWS Batch for compute scheduling may be a better option.
+Funnel will, by default, try to automatically load credentials from the environment. Alternatively, you may explicitly set the credentials in the config.
+
+Available Config:
+```yaml
+Database: dynamodb
+
+DynamoDB:
+ # Basename to use for dynamodb tables
+ TableBasename: "funnel"
+ # AWS region
+ Region: "us-west-2"
+ # AWS Access key ID
+ Key: ""
+ # AWS Secret Access Key
+ Secret: ""
+```
+
+### Known issues
+
+Dynamo does not store scheduler data. See [issue 340](https://github.com/ohsu-comp-bio/funnel/issues/340).
diff --git a/docs/tools/funnel/docs/databases/elasticsearch.md b/docs/tools/funnel/docs/databases/elasticsearch.md
new file mode 100644
index 0000000..e397348
--- /dev/null
+++ b/docs/tools/funnel/docs/databases/elasticsearch.md
@@ -0,0 +1,30 @@
+---
+title: Elasticsearch
+menu:
+ main:
+ parent: Databases
+---
+
+# Elasticsearch
+
+Funnel supports storing tasks and scheduler data in Elasticsearch (v8).
+
+Config:
+```yaml
+Database: elastic
+
+Elastic:
+ # Prefix to use for indexes
+ IndexPrefix: "funnel"
+ URL: http://localhost:9200
+ # Optional. Username for HTTP Basic Authentication.
+ Username:
+ # Optional. Password for HTTP Basic Authentication.
+ Password:
+ # Optional. Endpoint for the Elastic Service (https://elastic.co/cloud).
+ CloudID:
+ # Optional. Base64-encoded token for authorization; if set, overrides username/password and service token.
+ APIKey:
+ # Optional. Service token for authorization; if set, overrides username/password.
+ ServiceToken:
+```
diff --git a/docs/tools/funnel/docs/databases/mongodb.md b/docs/tools/funnel/docs/databases/mongodb.md
new file mode 100644
index 0000000..4a6e8ab
--- /dev/null
+++ b/docs/tools/funnel/docs/databases/mongodb.md
@@ -0,0 +1,24 @@
+---
+title: MongoDB
+menu:
+ main:
+ parent: Databases
+---
+
+# MongoDB
+
+Funnel supports storing tasks and scheduler data in MongoDB.
+
+Config:
+```yaml
+Database: mongodb
+
+MongoDB:
+ # Addresses for the seed servers.
+ Addrs:
+ - "localhost"
+ # Database name used within MongoDB to store funnel data.
+ Database: "funnel"
+ Username: ""
+ Password: ""
+```
diff --git a/docs/tools/funnel/docs/development.md b/docs/tools/funnel/docs/development.md
new file mode 100644
index 0000000..1412f0d
--- /dev/null
+++ b/docs/tools/funnel/docs/development.md
@@ -0,0 +1,8 @@
+---
+title: Development
+menu:
+ main:
+ weight: 30
+---
+
+# Development
diff --git a/docs/tools/funnel/docs/development/developers.md b/docs/tools/funnel/docs/development/developers.md
new file mode 100644
index 0000000..2d19f0a
--- /dev/null
+++ b/docs/tools/funnel/docs/development/developers.md
@@ -0,0 +1,97 @@
+---
+title: Funnel Developers
+
+menu:
+ main:
+ parent: Development
+ weight: 30
+---
+
+# Developers
+
+This page contains a rough collection of notes for people wanting to build Funnel from source and/or edit the code.
+
+### Building the Funnel source
+
+1. Install [Go 1.21+][go]. Check the version with `go version`.
+2. Ensure GOPATH is set. See [the docs][gopath] for help. Also, you probably want to add `$GOPATH/bin` to your `PATH`.
+3. Clone funnel and build
+
+ ```shell
+ $ git clone https://github.com/ohsu-comp-bio/funnel.git
+ $ cd funnel
+ $ make
+ ```
+
+4. Funnel is now downloaded and installed. Try `funnel version`.
+5. You can edit the code and run `make install` to recompile.
+
+### Developer Tools
+
+A Funnel development environment includes:
+
+- [Go 1.21+][go] for the majority of the code.
+- [Task Execution Schemas][tes] for task APIs.
+- [Protobuf][protobuf] + [gRPC][grpc] for RPC communication.
+- [gRPC Gateway][gateway] for HTTP communication.
+- [Angular][angular] and [SASS][sass] for the web dashboard.
+- [GNU Make][make] for development tasks.
+- [Docker][docker] for executing task containers (tested with v1.12, v1.13).
+- [dep][dep] for Go dependency vendoring.
+- [Make][make] for development/build commands.
+- [NodeJS][node] and [NPM][npm] for web dashboard development.
+
+### Makefile
+
+Most development tasks are run through `make` commands, including build, release, testing, website docs, lint, tidy, webdash dev, and more. See the [Makefile](https://github.com/ohsu-comp-bio/funnel/blob/master/Makefile) for an up-to-date list of commands.
+
+### Go Tests
+
+Run all tests: `make test`
+Run the worker tests: `go test ./worker/...`
+Run the worker tests with "Cancel" in the name: `go test ./worker -run Cancel`
+
+You get the idea. See the `go test` docs for more.
+
+### Mocking
+
+The [testify][testify] and [mockery][mockery] tools are used to generate and use
+mock interfaces in test code, for example, to mock the Google Cloud APIs.
+
+[go]: https://golang.org
+[angular]: https://angularjs.org/
+[protobuf]: https://github.com/google/protobuf
+[grpc]: http://www.grpc.io/
+[sass]: http://sass-lang.com/
+[make]: https://www.gnu.org/software/make/
+[docker]: https://docker.io
+[python]: https://www.python.org/
+[dep]: https://golang.github.io/dep/
+[node]: https://nodejs.org
+[npm]: https://www.npmjs.com/
+[gateway]: https://github.com/grpc-ecosystem/grpc-gateway
+[tes]: https://github.com/ga4gh/task-execution-schemas
+[testify]: https://github.com/stretchr/testify
+[mockery]: https://github.com/vektra/mockery
+[gopath]: https://golang.org/doc/code.html#GOPATH
+
+### Making a release
+
+- Update Makefile, edit `FUNNEL_VERSION` and `LAST_PR_NUMBER`
+ - `LAST_PR_NUMBER` can be found by looking at the previous release notes
+ from the previous release.
+- Run `make website`, which updates the download links and other content.
+ - Check the website locally by running `make website-dev`
+- Commit these changes.
+ - Because goreleaser requires a clean working tree in git
+ - This is a special case where it's easiest to commit to master.
+- Create a git tag: `git tag X.Y.Z`
+- Run `make release`
+ - This will build cross-platform binaries, build release notes,
+ and draft an unpublished GitHub release.
+ - Check the built artifacts by downloading the tarballs from the GitHub draft release
+ and running `funnel version`.
+- `git push origin master` to push your website and release changes.
+- A tagged docker image for the release will be built automatically on [dockerhub](https://hub.docker.com/repository/docker/quay.io/ohsu-comp-bio/funnel).
+- Publish the draft release on GitHub.
+- Copy `build/release/funnel.rb` to the `ohsu-comp-bio/homebrew-formula/Formula/funnel.rb` Homebrew formula repo, and push those changes to master.
diff --git a/docs/tools/funnel/docs/events.md b/docs/tools/funnel/docs/events.md
new file mode 100644
index 0000000..87941ef
--- /dev/null
+++ b/docs/tools/funnel/docs/events.md
@@ -0,0 +1,7 @@
+---
+title: Events
+menu:
+ main:
+ weight: 5
+---
+# Events
diff --git a/docs/tools/funnel/docs/events/kafka.md b/docs/tools/funnel/docs/events/kafka.md
new file mode 100644
index 0000000..242ce93
--- /dev/null
+++ b/docs/tools/funnel/docs/events/kafka.md
@@ -0,0 +1,22 @@
+---
+title: Kafka
+menu:
+ main:
+ parent: Events
+---
+
+# Kafka
+
+Funnel supports writing task events to a Kafka topic. To use this, add an event
+writer to the config:
+
+```yaml
+EventWriters:
+ - kafka
+ - log
+
+Kafka:
+ Servers:
+ - localhost:9092
+ Topic: funnel-events
+```
diff --git a/docs/tools/funnel/docs/integrations/nextflow.md b/docs/tools/funnel/docs/integrations/nextflow.md
new file mode 100644
index 0000000..3090a94
--- /dev/null
+++ b/docs/tools/funnel/docs/integrations/nextflow.md
@@ -0,0 +1,100 @@
+---
+title: Nextflow
+menu:
+ main:
+ parent: Integrations
+---
+
+> ⚠️ Nextflow support is currently in development and requires a few additional steps to run which are included below.
+
+# Nextflow
+
+[Nextflow](https://nextflow.io/) is a workflow engine with a [rich ecosystem]() of pipelines centered around biological analysis.
+
+> Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages.
+
+> Its fluent DSL simplifies the implementation and the deployment of complex parallel and reactive workflows on clouds and clusters.
+
+Since Nextflow [includes support](https://www.nextflow.io/docs/latest/executor.html#ga4gh-tes) for the TES API, it can be used in conjunction with Funnel to run tasks or to interact with a common TES endpoint.
+
+## Getting Started
+
+To set up Nextflow to use Funnel as the TES executor, run the following steps:
+
+### 1. Install Nextflow
+
+*Adapted from the [Nextflow Documentation](https://nextflow.io/docs/latest/install.html)*
+
+#### a. Install Nextflow:
+
+```sh
+curl -s https://get.nextflow.io | bash
+```
+
+This will create the nextflow executable in the current directory.
+
+#### b. Make Nextflow executable:
+
+```sh
+chmod +x nextflow
+```
+
+#### c. Move Nextflow into an executable path:
+
+```sh
+sudo mv nextflow /usr/local/bin
+```
+
+#### d. Confirm that Nextflow is installed correctly:
+
+```sh
+nextflow info
+```
+
+### 2. Update Nextflow Config
+
+Add the following to your `nextflow.config` in order to use the GA4GH TES plugin:
+
+```yaml
+cat <Processes all staged LFS files
Creates DRS records for new files
Only processes files that don't already exist on the DRS server
Prepares metadata for later upload during push | +| **Custom Transfer (upload)** | `git drs transfer` | Handles upload operations during `git push`
Creates indexd record on DRS server
Uploads file to Gen3-registered S3 bucket
Updates DRS object with access URLs | +| **Custom Transfer (download)** | `git drs transfer` | Handles download operations during `git lfs pull`
Retrieves file metadata from DRS server
Downloads file from configured storage
Validates checksums | + + +### Protocol Communication + +Git LFS and Git DRS communicate via JSON messages. Git LFS uses custom transfers to communicate with Git DRS, passing information through JSON protocol: + +```json +{ + "event": "init", + "operation": "upload", + "remote": "origin", + "concurrent": 3, + "concurrenttransfers": 3 +} +``` + +Response handling and logging occurs in transfer clients to avoid interfering with Git LFS stdout expectations. + +For more details, see the [Git LFS Custom Transfer Protocol](https://github.com/git-lfs/git-lfs/blob/main/docs/custom-transfers.md) documentation. + +## Configuration System + +Git DRS stores configuration in Git's local config (`.git/config`). + +**Example Configuration:** + +```bash +$ git config --list | grep drs +lfs.standalonetransferagent=drs +lfs.customtransfer.drs.args=transfer +lfs.customtransfer.drs.concurrent=true +lfs.customtransfer.drs.path=git-drs +lfs.customtransfer.drs.default-remote=calypr-public +lfs.customtransfer.drs.remote.calypr-dev.type=gen3 +lfs.customtransfer.drs.remote.calypr-dev.endpoint=https://calypr-public.ohsu.edu +lfs.customtransfer.drs.remote.calypr-dev.project=program-project +lfs.customtransfer.drs.remote.calypr-dev.bucket=my-bucket + +``` + +## Further Reading + +- [Git DRS Quick Start](quickstart.md) -- User guide for getting started +- [Troubleshooting](troubleshooting.md) -- Common issues and solutions +- [Git LFS Custom Transfer Agents](https://github.com/git-lfs/git-lfs/blob/main/docs/custom-transfers.md) -- Understanding the transfer protocol +- [Git LFS Specification](https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md) -- Pointer file format details +- [Git Hooks Documentation](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) -- Understanding Git hooks diff --git a/docs/tools/git-drs/index.md b/docs/tools/git-drs/index.md new file mode 100644 index 0000000..a32c2ae --- /dev/null +++ b/docs/tools/git-drs/index.md @@ -0,0 +1,69 @@ +--- +title: Git-DRS +--- + +# Git-DRS + +Git-DRS is a Git extension for managing large files in a Gen3 Data Commons using the **Data Repository Service (DRS)** content-addressable storage model. It essentially serves as a Git-LFS (Large File Storage) replacement tailored for Gen3. + +It allows you to: +- Track large files in your git repository without bloating it. +- Store the actual file contents in a Gen3 data commons (indexed via DRS). +- Seamlessly synchronize files between your local environment and the commons. + +## Installation + +Ensure `git-drs` is installed and in your PATH. + +## Initialization + +Initialize a repository to use Git-DRS. This sets up the necessary hooks and configuration. + +```bash +git-drs init +``` + +## Basic Workflow + +1. **Add a file**: Track a large file with Git-DRS. + ```bash + git-drs add
`git drs remote add`
`git drs remote list`
`git drs fetch`
`git drs push` | Repository and remote configuration
Setting up a new repository
Adding/managing DRS remotes
Refreshing expired credentials
Cross-remote promotion | +| **Git LFS** | `git lfs track`
`git lfs ls-files`
`git lfs pull`
`git lfs untrack` | File tracking and management
Defining which files to track
Downloading specific files
Checking file localization status | +| **Standard Git** | `git add`
`git commit`
`git push`
`git pull` | Version control operations
Normal development workflow
Git DRS runs automatically in background | + +## Authentication Errors + +### Error: `Upload error: 403 Forbidden` or `401 Unauthorized` + +**Cause**: Expired or invalid credentials + +**Solution**: + +```bash +# Download new credentials from your data commons +# Then refresh them by re-adding the remote +git drs remote add gen3 production \ + --cred /path/to/new-credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project my-project \ + --bucket my-bucket +``` + +**Prevention**: + +- Credentials expire after 30 days +- Set a reminder to refresh them regularly + +### Error: `Upload error: 503 Service Unavailable` + +**Cause**: DRS server is temporarily unavailable or credentials expired + +**Solutions**: + +1. Wait and retry the operation +2. Refresh credentials: + ```bash + git drs remote add gen3 production \ + --cred /path/to/credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project my-project \ + --bucket my-bucket + ``` +3. If persistent, download new credentials from the data commons + +## Network Errors + +### Error: `net/http: TLS handshake timeout` + +**Cause**: Network connectivity issues + +**Solution**: + +- Simply retry the command +- These are usually temporary network issues + +### Error: Git push timeout during large file uploads + +**Cause**: Long-running operations timing out + +**Solution**: Add to `~/.ssh/config`: + +``` +Host github.com + TCPKeepAlive yes + ServerAliveInterval 30 +``` + +## File Tracking Issues + +### Files Not Being Tracked by LFS + +**Symptoms**: + +- Large files committed directly to Git +- `git lfs ls-files` doesn't show your files + +**Solution**: + +```bash +# Check what's currently tracked +git lfs track + +# Track your file type +git lfs track "*.bam" +git add .gitattributes + +# Remove from Git and re-add +git rm --cached large-file.bam +git add large-file.bam +git commit -m "Track large file with LFS" +``` + +### Error: `[404] Object does not exist on the server` + +**Symptoms**: + +- After clone, git pull fails + +**Solution**: + +```bash +# Confirm repo has complete configuration +git drs remote list + +# Initialize your git drs project +git drs init + +# Add remote configuration +git drs remote add gen3 production \ + --cred /path/to/credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project my-project \ + --bucket my-bucket + +# Attempt git pull again +git lfs pull -I path/to/file +``` + +### Files Won't Download + +**Cause**: Files may not have been properly uploaded or DRS records missing + +**Solution**: + +```bash +# Check repository status +git drs remote list + +# Try pulling with verbose output +git lfs pull -I "problematic-file*" --verbose + +# Check logs +cat .git/drs/*.log +``` + +## Configuration Issues + +### Empty or Incomplete Configuration + +**Error**: `git drs remote list` shows empty or incomplete configuration + +**Cause**: Repository not properly initialized or no remotes configured + +**Solution**: + +```bash +# Initialize repository if needed +git drs init + +# Add Gen3 remote +git drs remote add gen3 production \ + --cred /path/to/credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project my-project \ + --bucket my-bucket + +# Verify configuration +git drs remote list +``` + +### Configuration Exists but Commands Fail + +**Cause**: Mismatched configuration between global and local settings, or expired credentials + +**Solution**: + +```bash +# Check configuration +git drs remote list + +# Refresh credentials by re-adding the remote +git drs remote add gen3 production \ + --cred /path/to/new-credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project my-project \ + --bucket my-bucket +``` + +## Remote Configuration Issues + +### Error: `no default remote configured` + +**Cause**: Repository initialized but no remotes added yet + +**Solution**: + +```bash +# Add your first remote (automatically becomes default) +git drs remote add gen3 production \ + --cred /path/to/credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project my-project \ + --bucket my-bucket +``` + +### Error: `default remote 'X' not found` + +**Cause**: Default remote was deleted or configuration is corrupted + +**Solution**: + +```bash +# List available remotes +git drs remote list + +# Set a different remote as default +git drs remote set staging + +# Or add a new remote +git drs remote add gen3 production \ + --cred /path/to/credentials.json \ + --url https://calypr-public.ohsu.edu \ + --project my-project \ + --bucket my-bucket +``` + +### Commands Using Wrong Remote + +**Cause**: Default remote is not the one you want to use + +**Solution**: + +```bash +# Check current default +git drs remote list + +# Option 1: Change default remote +git drs remote set production + +# Option 2: Specify remote for single command +git drs push staging +git drs fetch production +``` + +## Undoing Changes + +### Untrack LFS Files + +If you accidentally tracked the wrong files: + +```bash +# See current tracking +git lfs track + +# Remove incorrect pattern +git lfs untrack "wrong-dir/**" + +# Add correct pattern +git lfs track "correct-dir/**" + +# Stage the changes +git add .gitattributes +git commit -m "Fix LFS tracking patterns" +``` + +### Undo Git Add + +Remove files from staging area: + +```bash +# Check what's staged +git status + +# Unstage specific files +git restore --staged file1.bam file2.bam + +# Unstage all files +git restore --staged . +``` + +### Undo Last Commit + +To retry a commit with different files: + +```bash +# Undo last commit, keep files in working directory +git reset --soft HEAD~1 + +# Or undo and unstage files +git reset HEAD~1 + +# Or completely undo commit and changes (BE CAREFUL!) +git reset --hard HEAD~1 +``` + +### Remove Files from LFS History + +If you committed large files directly to Git by mistake: + +```bash +# Remove from Git history (use carefully!) +git filter-branch --tree-filter 'rm -f large-file.dat' HEAD + +# Then track properly with LFS +git lfs track "*.dat" +git add .gitattributes +git add large-file.dat +git commit -m "Track large file with LFS" +``` + +## Diagnostic Commands + +### Check System Status + +```bash +# Git DRS version and help +git-drs version +git-drs --help + +# Configuration +git drs remote list + +# Repository status +git status +git lfs ls-files +``` + +### Test Connectivity + +```bash +# Test basic Git operations +git lfs pull --dry-run + +# Test DRS configuration +git drs remote list +``` + +### Log Analysis + +When reporting issues, include: + +```bash +# System information +git-drs version +git lfs version +git --version + +# Configuration +git drs remote list +``` + +## Prevention Best Practices + +1. **Refresh credentials regularly** -- Credentials expire after 30 days. Set a calendar reminder to download and configure new credentials before they expire. + +2. **Test in small batches** -- Don't commit hundreds of files at once. Start with a few files to ensure your configuration works correctly. + +3. **Verify tracking** -- Always check `git lfs ls-files` after adding files to ensure they're being tracked by LFS. + +4. **Use .gitignore** -- Prevent accidental commits of temporary files, build artifacts, and other files that shouldn't be in the repository. + +5. **Monitor repository size** -- Keep an eye on `.git` directory size. If it grows unexpectedly, you may have committed large files directly to Git instead of through LFS. + +## Getting Help + +For issues not covered in this guide: + +- Check the [Git DRS Quick Start](quickstart.md) for setup instructions +- Review the [Developer Guide](developer-guide.md) for advanced usage +- Consult the [Git LFS FAQ](https://github.com/git-lfs/git-lfs/wiki/FAQ) +- See [GitHub's Git LFS documentation](https://docs.github.com/en/repositories/working-with-files/managing-large-files) diff --git a/docs/tools/grip/clients.md b/docs/tools/grip/clients.md new file mode 100644 index 0000000..986755e --- /dev/null +++ b/docs/tools/grip/clients.md @@ -0,0 +1,141 @@ +--- +title: Client Library +menu: + main: + identifier: clients + weight: 25 +--- + + +# Getting Started + +GRIP has an API for making graph queries using structured data. Queries are defined using a series of step [operations](./queries/index.md). + +## Install the Python Client + +Available on [PyPI](https://pypi.org/project/gripql/). + +``` +pip install gripql +``` + +Or install the latest development version: + +``` +pip install "git+https://github.com/bmeg/grip.git#subdirectory=gripql/python" +``` + + +## Using the Python Client + +Let's go through the features currently supported in the python client. + +First, import the client and create a connection to an GRIP server: + +```python +import gripql +G = gripql.Connection("https://bmeg.io").graph("bmeg") +``` + +Some GRIP servers may require authorizaiton to access its API endpoints. The client can be configured to pass +authorization headers in its requests. + +```python +import gripql + +# Basic Auth Header - {'Authorization': 'Basic dGVzdDpwYXNzd29yZA=='} +G = gripql.Connection("https://bmeg.io", user="test", password="password").graph("bmeg") +# + +# Bearer Token - {'Authorization': 'Bearer iamnotarealtoken'} +G = gripql.Connection("https://bmeg.io", token="iamnotarealtoken").graph("bmeg") + +# OAuth2 / Custom - {"OauthEmail": "fake.user@gmail.com", "OauthAccessToken": "iamnotarealtoken", "OauthExpires": 1551985931} +G = gripql.Connection("https://bmeg.io", credential_file="~/.grip_token.json").graph("bmeg") +``` + +Now that we have a connection to a graph instance, we can use this to make all of our queries. + +One of the first things you probably want to do is find some vertex out of all of the vertexes available in the system. In order to do this, we need to know something about the vertex we are looking for. To start, let's see if we can find a specific gene: + +```python +result = G.V().hasLabel("Gene").has(gripql.eq("symbol", "TP53")).execute() +print(result) +``` + +A couple things about this first and simplest query. We start with `O`, our grip client instance connected to the "bmeg" graph, and create a new query with ``. This query is now being constructed. You can chain along as many operations as you want, and nothing will actually get sent to the server until you print the results. + +Once we make this query, we get a result: + +```python +[ + { + u'_id': u'ENSG00000141510', + u'_label': u'Gene' + u'end': 7687550, + u'description': u'tumor protein p53 [Source:HGNC Symbol%3BAcc:HGNC:11998]', + u'symbol': u'TP53', + u'start': 7661779, + u'seqId': u'17', + u'strand': u'-', + u'id': u'ENSG00000141510', + u'chromosome': u'17' + } +] +``` + +This represents the vertex we queried for above. All vertexes in the system will have a similar structure, basically: + +* _\_id_: This represents the global identifier for this vertex. In order to draw edges between different vertexes from different data sets we need an identifier that can be constructed from available data. Often, the `_id` will be the field that you query on as a starting point for a traversal. +* _\_label_: The label represents the type of the vertex. All vertexes with a given label will share many property keys and edge labels, and form a logical group within the system. + +The data on a query result can be accessed as properties on the result object; for example `result[0].data.symbol` would return: + +```python +u'TP53' +``` + +You can also do a `has` query with a list of items using `gripql.within([...])` (other conditions exist, see the `Conditions` section below): + +```python +result = G.V().hasLabel("Gene").has(gripql.within("symbol", ["TP53", "BRCA1"])).render({"_id": "_id", "symbol":"symbol"}).execute() +print(result) +``` + +This returns both Gene vertexes: + +``` +[ + {u'symbol': u'TP53', u'_id': u'ENSG00000141510'}, + {u'symbol': u'BRCA1', u'_id': u'ENSG00000012048'} +] +``` + +Once you are on a vertex, you can travel through that vertex's edges to find the vertexes it is connected to. Sometimes you don't even need to go all the way to the next vertex, the information on the edge between them may be sufficient. + +Edges in the graph are directional, so there are both incoming and outgoing edges from each vertex, leading to other vertexes in the graph. Edges also have a _label_, which distinguishes the kind of connections different vertexes can have with one another. + +Starting with gene TP53, and see what kind of other vertexes it is connected to. + +```python +result = G.V().hasLabel("Gene").has(gripql.eq("symbol", "TP53")).in_("TranscriptFor")render({"id": "_id", "label":"_label"}).execute() +print(result) +``` + +Here we have introduced a couple of new steps. The first is `.in_()`. This starts from wherever you are in the graph at the moment and travels out along all the incoming edges. +Additionally, we have provided `TranscriptFor` as an argument to `.in_()`. This limits the returned vertices to only those connected to the `Gene` verticies by edges labeled `TranscriptFor`. + + +``` +[ + {u'_label': u'Transcript', u'_id': u'ENST00000413465'}, + {u'_label': u'Transcript', u'_id': u'ENST00000604348'}, + ... +] +``` + +View a list of all available query operations [here](./queries/index.md). + +### Using the command line + +Grip command line syntax is defined at gripql/javascript/gripql.js diff --git a/docs/tools/grip/commands/create.md b/docs/tools/grip/commands/create.md new file mode 100644 index 0000000..bde4aa8 --- /dev/null +++ b/docs/tools/grip/commands/create.md @@ -0,0 +1,27 @@ + +--- +title: create + +menu: + main: + parent: commands + weight: 2 +--- + +# `create` + +## Usage + +```bash +gripql-cli create