programming.dev/psql perf updates

.github/CODEOWNERS

@@ -0,0 +1 @@
+* @Nutomic @dessalines @codyro @ticoombs

.woodpecker.yml

@@ -2,10 +2,16 @@ pipeline:
   prettier_markdown_check:
     image: tmknom/prettier
     commands:
-      - prettier -c "*.md" "*.yml"
+      - prettier -c "*.md" "*.yml" "examples/vars.yml"
   check_ansible_format:
     image: alpine:3
     commands:
       - apk add ansible
       - ansible-playbook lemmy.yml --syntax-check
+      - ansible-playbook lemmy-almalinux.yml --syntax-check
       - ansible-playbook uninstall.yml --syntax-check
+  ansible_lint:
+    image: alpine:3
+    commands:
+      - apk add ansible ansible-lint
+      - ansible-lint --warn-list experimental lemmy.yml lemmy-almalinux.yml uninstall.yml examples/vars.yml

README.md

@@ -1,23 +1,38 @@
 # Lemmy-Ansible
 
-This provides an easy way to install [Lemmy](https://github.com/LemmyNet/lemmy) on any server. It automatically sets up an nginx server, letsencrypt certificates, and email.
+This provides an easy way to install [Lemmy](https://github.com/LemmyNet/lemmy) on any server. It automatically sets up an nginx server, letsencrypt certificates, docker containers, pict-rs, and email smtp.
 
 ## Requirements
 
 To run this ansible playbook, you need to:
 
-- Have a Debian-based server / VPS where lemmy will run.
+- Have a Debian/AlmaLinux 9-based server / VPS where lemmy will run.
+- Supported CPU architectures are x86-64 and ARM64.
 - Configure a DNS `A` Record to point at your server's IP address.
 - Make sure you can ssh to it, with a sudo user: `ssh <your-user>@<your-domain>`
-- Install [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) on your **local** machine (do not install it on your destination server).
+- Install [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) (>= `2.11.0` on your **local** machine (do not install it on your destination server).
+
+### Supported Distribution Playbook Matrix
+
+These are the distributions we currently support. Anything not listed here is currently not supported.  
+If you wish to see another distribution on the list, please test on the latest commit in `main` and report your findings via an Issue.
+
+| Distribution | Version   | Playbook              |
+| ------------ | --------- | --------------------- |
+| Debian       | 10        | `lemmy.yml`           |
+| Debian       | 11        | `lemmy.yml`           |
+| Debian       | 12        | `lemmy.yml`           |
+| Ubuntu       | 22.04 LTS | `lemmy.yml`           |
+| RHEL         | 9         | `lemmy-almalinux.yml` |
 
 ## Install
 
-1. Clone this repo:
+1. Clone this repo & checkout latest tag
 
    ```
    git clone https://github.com/LemmyNet/lemmy-ansible.git
    cd lemmy-ansible
+   git checkout $(git describe --tags)
    ```
 
 2. Make a directory to hold your config:
@@ -44,7 +59,15 @@ To run this ansible playbook, you need to:
 
    You can use [the PGTune tool](https://pgtune.leopard.in.ua) to tune your postgres to meet your server memory and CPU.
 
-6. Run the playbook:
+6. Copy the sample `vars.yml` file
+
+   `cp examples/vars.yml inventory/host_vars/<your-domain>/vars.yml`
+
+   Edit the `inventory/host_vars/<your-domain>/vars.yml` file to your liking.
+
+7. Run the playbook:
+
+   _Note_: See the "Supported Distribution Playbook Matrix" section above if you should use `lemmy.yml` or not
 
    `ansible-playbook -i inventory/hosts lemmy.yml`
 
@@ -66,9 +89,85 @@ To run this ansible playbook, you need to:
 
 ## Upgrading
 
-- Run `git pull`
-- Check out the [Lemmy Releases Changelog](https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md) to see if there are any config changes with the releases since your last.
-- Run `ansible-playbook -i inventory/hosts lemmy.yml --become`
+Since version `1.1.0` we no longer default to using `main` but use tags to make sure deployments are versioned.
+With every new release all migration steps shall be written below so make sure you check out the [Lemmy Releases Changelog](https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md) to see if there are any config changes with the releases since your last read.
+
+### Upgrading to 1.3.0 (Lemmy 0.19.0 & pictrs-0.4.7)
+
+This is a major change and has required reading! tl;dr
+
+- Lemmy has been upgraded to 0.19.0
+- pict-rs has been upgraded to 0.4.7
+  - pict-rs has not been integrated with postgres yet
+- "Optional Modules" are now available to be added to your lemmy install as provided by the community.
+  - The first being pictrs-safety
+
+#### Steps
+
+- Prepare to have downtime as the database needs to perform migrations!
+- Run `git pull && git checkout 1.3.0`
+- Run your regular deployment. Example: `ansible-playbook -i inventory/hosts lemmy.yml --become`
+- Lemmy will now be down! In testing this takes from 20 to 60 minutes.
+  - If you are bored you can ssh into your server, and check the logs on postgres for updates
+  - `docker compose logs -f postgres` while ssh'd and in your Lemmy directory
+
+#### Update your pict-rs sled-database (Optional)
+
+If you are happy for pict-rs to be down _for a while_ go straight to our `1.3.1` git tag which updates pictrs to 0.5.0. Otherwise keep reading.
+Starting with 0.5.0 your database will automatically upgrade to the latest version, which will cause downtime for your users.
+As such there is an intermediary step where you can upgrade your database in the background to prepare for 0.5 (Reference documentation)[https://git.asonix.dog/asonix/pict-rs/releases#user-content-upgrade-preparation-endpoint]. This ensure no-one is caught out by unforseen downtime of multiple services.
+
+Once you have deployed lemmy-ansible `1.3.0` tag, please continue (if you want):
+
+- Take note of what your pict-rs API Key is under `vars.yml`
+- Take note of what your docker network name is. (It's normally the domain without any extra characters)
+  - You should be able to find it via: `docker network ls | grep _default` if in doubt.
+- Run the following command replacing `api-key` with the pict-rs api key, & `youdomain` with the network name.
+- `docker run --network yourdomain_default --rm curlimages/curl:8.5.0 --silent -XPOST -H'X-Api-Token: api-key' 'http://pictrs:8080/internal/prepare_upgrade'`
+- This will start the background process updating your database from 0.4 to 0.5 compatible.
+
+This is only Optional, and takes a shorter amount of time than the Lemmy database upgrade, but on huge installations it may take a lot longer.
+
+#### Optional Module(s)
+
+Our first optional module is [pictrs-safety](https://github.com/db0/pictrs-safety). See the repo linked for more information, especially for integration with pictrs (which is what it is for) Thanks to @db0 for their contribution.  
+See the `pictrs_safety_env_vars` under `examples/vars.yml` for relevant options (and the two password variables)  
+To enable this module to be used you must ADD `pictrs_safety: true` to your `vars.yml`.
+
+### Upgrading to 1.2.1 (Lemmy 0.18.5)
+
+This is a minor change which fixes the issue with the Postgres container not using the `customPostgres.conf` file.
+
+#### Steps
+
+- Please regenerate your `customPostgres.conf` from `examples/customPostgres.conf`
+- **OR**
+- Add the following block to your current customPostgres file.
+
+```
+# Listen beyond localhost
+listen_addresses = '*'
+```
+
+### Upgrading to 1.2.0 (Lemmy 0.18.5)
+
+Major changes:
+
+- All variables are not under a singular file so you will not need to modify anything: `inventory/host_vars/{{ domain }}/vars.yml`
+- `--become` is now optional instead of forced on
+
+#### Steps
+
+- Run `git pull && git checkout 1.2.0`
+- When upgrading from older versions of these playbooks, you will need to do the following:
+  - Rename `inventory/host_vars/{{ domain }}/passwords/postgres` file to `inventory/host_vars/{{ domain }}/passwords/postgres.psk`
+  - Copy the `examples/vars.yml` file to `inventory/host_vars/{{ domain }}/vars.yml`
+  - Edit your variables as desired
+- Run your regular deployment. Example: `ansible-playbook -i inventory/hosts lemmy.yml --become`
+
+### Upgrading to 1.1.0 (Lemmy 0.18.3)
+
+- No major changes should be required
 
 ## Migrating your existing install to use this deploy
 

VERSION

@@ -1 +1 @@
-0.18.3
+0.19.1

ansible.cfg

@@ -4,6 +4,3 @@ inventory = inventory
 
 [ssh_connection]
 pipelining = True
-
-[privilege_escalation]
-become = True

examples/customPostgresql.conf

@@ -24,7 +24,9 @@ max_parallel_workers = 4
 max_parallel_maintenance_workers = 2
 
 # Other custom params
-temp_file_size=1GB
 synchronous_commit=off
 # This one shouldn't be on regularly, because DB migrations often take a long time
 # statement_timeout = 10000
+
+# Listen beyond localhost
+listen_addresses = '*'

examples/hosts

@@ -1,14 +1,16 @@
 [lemmy]
 # to get started, copy this file to `inventory` and adjust the values below.
 # - `myuser@example.com`: replace with the destination you use to connect to your server via ssh
+# - `ansible_user=root`: replace `root` with your the username you use to connect to your ssh server
 # - `domain=example.com`: replace `example.com` with your lemmy domain
 # - `letsencrypt_contact_email=your@email.com` replace `your@email.com` with your email address,
 #                                              to get notifications if your ssl cert expires
 # - `lemmy_base_dir=/srv/lemmy`: the location on the server where lemmy can be installed, can be any folder
 #                                if you are upgrading from a previous version, set this to `/lemmy`
 # - `lemmy_version`: <Optional> The back end version.
 # - `lemmy_ui_version`: <Optional> overrides the front end version.
-myuser@example.com  domain=example.com  letsencrypt_contact_email=your@email.com  lemmy_base_dir=/srv/lemmy
+# - `pictrs_safety`: <Optional> If true, a docker container for pictrs-safety will be deployed and pict-rs will be configured to validate images through it. You will also need to set up a fedi-safety worker to validate the images.
+example.com  ansible_user=root domain=example.com  letsencrypt_contact_email=your@email.com  lemmy_base_dir=/srv/lemmy pictrs_safety=false
 
 [all:vars]
 ansible_connection=ssh

examples/vars.yml

@@ -0,0 +1,51 @@
+postgres_password: "{{ lookup('password', 'inventory/host_vars/{{ domain }}/passwords/postgres.psk chars=ascii_letters,digits') }}" # noqa yaml[line-length]:w
+
+# Next two only relevant if pictrs_safety == True
+pictrs_safety_worker_auth: "{{ lookup('password', 'inventory/host_vars/{{ domain }}/passwords/pictrs_safety_worker_auth.psk chars=ascii_letters,digits length=15') }}" # noqa yaml[line-length]
+pictrs_safety_secret: "{{ lookup('password', 'inventory/host_vars/{{ domain }}/passwords/pictrs_safety_secret.psk chars=ascii_letters,digits length=80') }}" # noqa yaml[line-length]
+
+# You can set any pict-rs environmental variables here. They will populate the templates/docker-compose.yml file.
+# https://git.asonix.dog/asonix/pict-rs
+pictrs_env_vars:
+  - PICTRS__SERVER__API_KEY: "{{ postgres_password }}"
+  - PICTRS__MEDIA__VIDEO_CODEC: vp9
+  - PICTRS__MEDIA__GIF__MAX_WIDTH: 256
+  - PICTRS__MEDIA__GIF__MAX_HEIGHT: 256
+  - PICTRS__MEDIA__GIF__MAX_AREA: 65536
+  - PICTRS__MEDIA__GIF__MAX_FRAME_COUNT: 400
+  - PICTRS_OPENTELEMETRY_URL: http://otel:4137
+  - RUST_LOG: debug
+  - RUST_BACKTRACE: full
+#  - PICTRS__STORE__TYPE: object_storage
+#  - PICTRS__STORE__ENDPOINT: '<S3 endpoint>'
+#  - PICTRS__STORE__BUCKET_NAME: '<bucket name>'
+#  - PICTRS__STORE__REGION: '<region>'
+#  - PICTRS__STORE__USE_PATH_STYLE: false
+#  - PICTRS__STORE__ACCESS_KEY: '<access key>'
+#  - PICTRS__STORE__SECRET_KEY: '<secret key>'
+
+postgres_env_vars:
+  - POSTGRES_USER: lemmy
+  - POSTGRES_PASSWORD: "{{ postgres_password }}"
+  - POSTGRES_DB: lemmy
+
+lemmy_env_vars:
+  - RUST_LOG: warn
+  - MAX_DB_CONNECTIONS=20
+
+lemmyui_env_vars:
+  - LEMMY_UI_LEMMY_INTERNAL_HOST: lemmy:8536
+  - LEMMY_UI_LEMMY_EXTERNAL_HOST: "{{ domain }}"
+  - LEMMY_UI_HTTPS: true
+
+postfix_env_vars:
+  - POSTFIX_myhostname: "{{ domain }}"
+
+pictrs_safety_env_vars:
+  # Use this in your fedi-safety to allow your worker to authenticate to pictrs-safety
+  - FEDIVERSE_SAFETY_WORKER_AUTH: "{{ pictrs_safety_worker_auth }}"
+  - FEDIVERSE_SAFETY_IMGDIR: "/tmp/images"
+  - USE_SQLITE: 1
+  - secret_key: "{{ pictrs_safety_secret }}"
+  - SCAN_BYPASS_THRESHOLD: 10
+  - MISSING_WORKER_THRESHOLD: 5

files/proxy_params

@@ -0,0 +1,4 @@
+proxy_set_header Host $http_host;
+proxy_set_header X-Real-IP $remote_addr;
+proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+proxy_set_header X-Forwarded-Proto $scheme;