Skip to content

README.md

Latest commit

 

History

History
36 lines (25 loc) · 2.15 KB

README.md

File metadata and controls

36 lines (25 loc) · 2.15 KB

data

Running Locally:

  • Use Python 3.11

    • You can use a docker container to get the correct version of python: docker run -it --rm --volume .:/app --workdir /app --volume ~/.prefect:/root/.prefect python:3.11 bash

  • (Only for first time setup) Create a virtual environment: python -m venv env

  • Enter your virtual environment: source ./env/bin/activate

  • (Only for first time setup) Install dependencies pip install -r requirements.txt

  • Ensure you have RolyPoly credentials

    • Use the /set-password command in the #commands channel of the Sprocket Discord server if you need to set a password
    • Can check and reset your password using the /test-login and /update-password commands respectively if you don't remember your password

  • Configure Prefect to talk to Sprocket Prefect

    • prefect config set PREFECT_API_URL="https://[discord username]:[rolypoly password]@prefect.spr.ocket.dev/api"
    • Note that if you have special characters in your password you will need to URL encode them
      • For example, a ! will become %21

  • Test connection with: prefect deployment ls

    • There should be at least one item here

  • Now you can test all runs with: python ./src/flows/process-query-directory.py

Dev database snapshots

  • Create a dev snapshot from GitHub Actions by manually running Create Dev Database Snapshot. It uses the existing PGPASS_CONTENT and AWS_CREDS_CONTENT secrets, writes a timestamped PostgreSQL custom-format dump, and updates dev-snapshots/postgres/latest.dump in the datasets Spaces bucket.
  • To import from Spaces, install the AWS CLI and provide Spaces credentials via your normal AWS environment or --aws-creds-file.
  • Import the latest snapshot locally with Docker Postgres:
    • scripts/import-dev-snapshot.sh --container datasets-postgres --dbname datasets --user postgres --clean --create-db
  • Or import into a local Postgres connection with pg_restore installed:
    • scripts/import-dev-snapshot.sh --host localhost --port 5432 --dbname datasets --user postgres --clean --create-db