Skip to content

getting-started.md

Latest commit

 

History

History
373 lines (259 loc) · 7.85 KB

getting-started.md

File metadata and controls

373 lines (259 loc) · 7.85 KB

Getting Started

This guide walks you through setting up Git DRS and performing common workflows.

Navigation: Installation → Getting Started → Commands Reference → Troubleshooting

Repository Initialization

Every Git repository using Git DRS requires configuration, whether you're creating a new repo or cloning an existing one.

Cloning Existing Repository (Gen3)

  1. Clone the Repository

    git clone <repo-clone-url>.git
cd <name-of-repo>

  2. Configure SSH (if using SSH URLs)

    If using SSH URLs like git@github.com:user/repo.git, add to ~/.ssh/config:

    Host github.com
    TCPKeepAlive yes
    ServerAliveInterval 30

  3. Get Credentials

  4. Initialize Repository

    git drs init

  5. Verify Configuration

    git drs remote list

    Output:

    * production  gen3    https://calypr-public.ohsu.edu/

    The * indicates this is the default remote.

New Repository Setup (Gen3)

  1. Create and Clone Repository

    git clone <repo-clone-url>.git
cd <name-of-repo>

  2. Configure SSH (if needed - same as above)

  3. Get Credentials (same as above)

  4. Get Project Details

    Contact your data coordinator for:

    • DRS server URL
    • Organization name
    • Project ID
    • Bucket name
    • Confirmation that bucket mapping exists for your organization/project

  5. Initialize Git DRS

    git drs init

  6. Add Remote Configuration

    git drs remote add gen3 production \
    --cred /path/to/credentials.json \
    --url https://calypr-public.ohsu.edu \
    --project my-project \
    --bucket my-bucket

    Note: Since this is your first remote, it automatically becomes the default. No need to run git drs remote set.

  7. Verify Configuration

    git drs remote list

    Output:

    * production  gen3    https://calypr-public.ohsu.edu

    Important: git drs remote add alone is not enough. Push/pull requires an existing bucket mapping for your organization/project (usually provisioned once by a steward/admin).

Managing Additional Remotes

You can add more remotes later for multi-environment workflows (development, staging, production):

# Add staging remote
git drs remote add gen3 staging \
    --cred /path/to/staging-credentials.json \
    --url https://staging.calypr.ohsu.edu \
    --project staging-project \
    --bucket staging-bucket

# View all remotes
git drs remote list

# Switch default remote
git drs remote set staging

# Or use specific remote for one command
git drs push production
git drs fetch staging

File Tracking

Git DRS can use Git LFS-compatible pointers and local object storage. You must explicitly track file patterns before adding LFS-managed files.

View Current Tracking

git lfs track

Track Files

Single File

git lfs track path/to/specific-file.txt
git add .gitattributes

File Pattern

git lfs track "*.bam"
git add .gitattributes

Directory

git lfs track "data/**"
git add .gitattributes

Untrack Files

# View tracked patterns
git lfs track

# Remove pattern
git lfs untrack "*.bam"

# Stage changes
git add .gitattributes

Basic Workflows

Adding and Pushing Files

# Track file type (if not already tracked)
git lfs track "*.bam"
git add .gitattributes

# Add your file
git add myfile.bam

# Verify LFS is tracking it
git lfs ls-files

# Commit and push
git commit -m "Add new data file"
git push

Note: Git DRS automatically creates DRS records during commit and uploads files to the default remote during push.

Downloading Files

Single File

git lfs pull -I path/to/file.bam

Pattern

git lfs pull -I "*.bam"

All Files

git lfs pull

Directory

git lfs pull -I "data/**"

Checking File Status

# List all LFS-tracked files
git lfs ls-files

# Check specific pattern
git lfs ls-files -I "*.bam"

# View localization status
# (-) = not localized, (*) = localized
git lfs ls-files

Working with Cloud Object URLs

You can add references to existing bucket objects without copying them:

# Track the file pattern first
git lfs track "myfile.txt"
git add .gitattributes

# Add object reference (known sha256 path)
git drs add-url s3://bucket/path/to/file \
  --sha256 <file-hash>

# Or use unknown-sha (experimental sentinel mode)
git drs add-url s3://bucket/path/to/file

# Commit and push
git commit -m "Add S3 file reference"
git push

See Cloud URL Integration Guide for detailed examples.

Configuration Management

View Configuration

git drs remote list

Update Configuration

# Refresh credentials - re-add remote with new credentials
git drs remote add gen3 production \
    --cred /path/to/new-credentials.json \
    --url https://calypr-public.ohsu.edu \
    --project my-project \
    --bucket my-bucket

# Switch default remote
git drs remote set staging

View Logs

  • Logs location: .git/drs/ directory

Command Summary

Action Commands
Initialize git drs init
Add remote git drs remote add gen3 <name> --cred...
View remotes git drs remote list
Set default git drs remote set <name>
Track files git lfs track "pattern"
Check tracked git lfs ls-files
Add files git add file.ext
Commit git commit -m "message"
Push git push
Download git lfs pull -I "pattern"

Session Workflow

Note: You do NOT need to run git drs init again. Initialization is a one-time setup per Git repository clone.

For each work session:

  1. Refresh credentials (if expired - credentials expire after 30 days)

    git drs remote add gen3 production \
    --cred /path/to/new-credentials.json \
    --url https://calypr-public.ohsu.edu \
    --project my-project \
    --bucket my-bucket

  2. Work with files (track, add, commit, push)

Local DRS Server Setup

Use this flow when developing against a local drs-server instead of hosted Gen3.

  1. Initialize repo

    git drs init

  2. Add local remote

    git drs remote add local origin http://localhost:8080 \
    --organization calypr \
    --project end_to_end_test \
    --bucket cbds \
    --username drs-user \
    --password drs-pass

    If your local server has no basic auth, omit --username/--password.

  3. Track and push

    git lfs track "*.bin"
git add .gitattributes data/example.bin
git commit -m "Add local DRS test file"
git drs push

  4. Verify pull

    git drs pull
# or the Git LFS compatibility path
git lfs pull

For complete local/remote mode behavior and e2e runbooks, see E2E Modes + Local Setup.

  1. Download files as needed

    git lfs pull -I "required-files*"

Next Steps