Add offline backup for foremanctl

[sjha4]

Implements comprehensive offline backup functionality for Foreman deployments:

• Backs up all databases (foreman, candlepin, pulp, 5 IOP DBs)
• Backs up foremanctl state
• Includes metadata with container image digests for restore compatibility
• Preflight checks for running tasks and database integrity (amcheck)
• Automatic service restoration on failure

Why are you introducing these changes? (Problem description, related links)

What are the changes introduced in this pull request?

• Offline backup

How to test this pull request

I got a foremanctl box with normal deploy. On this box, clone foremanctl repo and checkout this branch.
cd /root/foremanctl
source .venv/bin/activate
export OBSAH_STATE=/var/lib/foremanctl

Then try ./foremanctl --help

Also,

(.venv) [root@ip-10-0-167-40 foremanctl]# ./foremanctl backup  --help
usage: foremanctl backup [-h] [-v] [--incremental] [--online] [--skip-pulp-content] [--tar-volume-size TAR_VOLUME_SIZE] [--wait-for-tasks]
                         backup_dir

Create offline backup of Foreman databases and configuration

positional arguments:
  backup_dir            Directory where backup files will be stored

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output
  --incremental         Perform incremental backup (not yet implemented)
  --online              Perform online backup without stopping services (not yet implemented)
  --skip-pulp-content   Skip Pulp content directory backup (not yet implemented)
  --tar-volume-size TAR_VOLUME_SIZE
                        Split tar archives at specified size in MB (not yet implemented, for Pulp content only)
  --wait-for-tasks      Wait for running tasks to complete instead of failing immediately

The help section has placeholders for incremental, online and --tar-volume-size which will be implemented in follow up cards/PRs.

You can run :
Command:

cd /root/foremanctl
source .venv/bin/activate
export OBSAH_STATE=/var/lib/foremanctl
./foremanctl backup /var/tmp/foreman-backup-test --wait-for-tasks

I have a dummy restore script which can be used for testing and also getting steps to run manually when testing.:

Download the script

wget https://gist.githubusercontent.com/sjha4/35d98b318f15753a678a406fb0fb14ad/raw/test-restore-final.sh

Make it executable

chmod +x test-restore-final.sh

Run restore

./test-restore-final.sh /path/to/backup/foreman-backup-TIMESTAMP

Steps to reproduce:

• On a foremanctl box, run foremanctl backup BACKUP_DIR

Checklist

• Tests added/updated (if applicable)
• Documentation updated (if applicable)

[ianballou]

I did an automated test run, take these with a grain of salt, but I wanted to post early in case they're real for extra time to test and fix.

Firstly, the DBs did get backed up, so awesome, but there were some hiccups that stopped the full process from running:

Bug #1: podman_network.yaml fails when no custom networks exist

File: src/playbooks/backup/tasks/podman_network.yaml
Severity: Blocker — prevents backup from completing
Description: The shell command podman network ls --format '{{.Name}}' | grep -v '^podman$' | while read net; do ... returns exit code 1 when there are no custom networks, because grep -v finds no matching lines.
Fix: Add || true after the grep, or use a different approach:

# Option A: tolerate empty result
  failed_when: networks_json.rc not in [0, 1]

# Option B: check first, skip if no custom networks

Bug #2: Wrong Foreman tasks API endpoint

File: src/playbooks/backup/tasks/preflight.yaml
Severity: High — preflight silently skips running task detection
Description: Uses https://{{ fqdn }}/api/v2/tasks?state=running which returns 404. The correct endpoint is https://{{ fqdn }}/foreman_tasks/api/tasks?state=running&search=state%3Drunning. Because failed_when: false is set, the error is silently ignored.
Impact: Backups will proceed even with running Foreman tasks, risking data inconsistency.

Bug #3: pg_isready and pg_dump not available on host

... I cut the output here, I'm not sure why these commands weren't on my box. It's not related to this PR I don't think.

Bug #4: Hardcoded parameters.yaml path in metadata task

File: src/playbooks/backup/tasks/metadata.yaml
Severity: Low — affects metadata accuracy only
Description: ansible.builtin.slurp reads from /var/lib/foremanctl/parameters.yaml but foremanctl's state directory is configurable via OBSAH_STATE. In dev/vagrant setups, the actual path is different (e.g., /vagrant/.var/lib/foremanctl/parameters.yaml).
Impact: enabled_features: [] in metadata despite features being configured. Doesn't affect DB dump correctness.
Fix: Use the state_dir variable instead of hardcoding the path.

[sjha4]

Will update the tasks endpoint and parameters.yaml path..

About the podman networks, those are created for IOP.. Like https://github.com/theforeman/foremanctl/blob/master/src/roles/iop_network/tasks/main.yaml so I'd assume we have that present in production deployments. We can add some handling for when it's not.

[aidenfine]

Maybe nitpick but does it make sense to include the not yet implemented flags when doing backup --help? Would you be opposed to just leaving them out in this PR?

[sjha4]

Maybe nitpick but does it make sense to include the not yet implemented flags when doing backup --help? Would you be opposed to just leaving them out in this PR?

I am fine either way but it's helpful guidance for future PRs and documentation to look at.

[evgeni]

not a full review (I stopped somewhere around the secrets backup), but overall this feels a lot like "let's write a huge bash script and then wrap it in YAML" and not like Ansible :(

[ehelms]

If these are all in service of writing the metadata file, you should be able to set them as vars on the tasks:

- name: Write metadata file
  ansible.builtin.copy:
    content: "{{ backup_metadata | to_nice_yaml }}"
    dest: "{{ backup_dir_full }}/metadata.yml"
    mode: '0644'
  vars:
     backup_metadata:
       hostname: "{{ ansible_fqdn }}"
     ...

You could also consider metadata.yml being a template.

[sjha4]

Updated as vars for the task..

[ehelms]

This is tricky because you are bleeding other role variables into this role which goes against our design principle.

[sjha4]

Moved these to backup_* specific variables in base.yml following pattern for other roles..

[sjha4]

Should be getting close here..Addressed prior reviews..2 pending reviews around using pulp API instead of pulp db queries for running tasks and documentation for backup..

[sjha4]

@evgeni @ehelms This should be good to review..On the documentation part, I am leaning towards adding some documentation with the restore part of this which would be the complete workflow. The restore piece is being worked on now and we will add the backup-restore workflow documentation with it.

[ianballou]

I don't think it's a blocker, but we could use pulp.squeezer here https://galaxy.ansible.com/ui/repo/published/pulp/squeezer/content/module/task/?keywords=task

[ianballou]

I think it makes sense to always use --listed-incremental with tar here so that the incremental metadata is available from the start even before we fully support incremental backup.

Foreman maintain backup passes the incremental flag to tar regardless of if the incremental flag was passed to foreman maintain: https://github.com/theforeman/foreman_maintain/blob/344d337491072e6bbed1b7987284d8f41889cbf9/definitions/procedures/backup/pulp.rb#L36-L44

When we do add support for incremental backups, then we can copy around the .snar files and use them properly.

[sjha4]

It is interconnected as in you'd need a full backup as level 0 backup and the snar and then move to incremental on top of it. Implementation wise, I'd propose getting there as part of incremental and scoping this PR to full backups only?

[sjha4]

@ehelms Added user documentation for backup..We can reuse and update the file as we develop restore command and expand the backup feature.

I was not sure if we needed any developer documentation for this or backup/restore in general.