parallell rsyncs

[alfs]

If the repo is down, nodes end up with multiple rsync processes trying to upload the results. E.g.

monroe@Monroe000db945e338:~$ ps auxf|grep -i rsync|wc -l
67
monroe@Monroe000db945e338:~$ ps auxf|grep -i rsync|tail
root     32132  0.0  0.0  11228  3120 ?        S    09:24   0:00  |           \_ /bin/bash /usr/bin/monroe-rsync-results
root     32183  0.0  0.0  13084  3468 ?        S    09:24   0:00  |               \_ rsync -r -t -z --exclude tstat/ --include */ --include *.json --include *.xz --include *.gz --exclude * --prune-empty-dirs --chmod=644 -e ssh -i /etc/keys/repo.monroe-system.eu -p 2280 -oStrictHostKeyChecking=no -o ConnectTimeout=30 --remove-source-files --timeout=30 /experiments/monroe/ rsync@repo.monroe-system.eu:/experiments/monroe
root     32184  0.0  0.1  47508  6012 ?        S    09:24   0:00  |                   \_ ssh -i /etc/keys/repo.monroe-system.eu -p 2280 -oStrictHostKeyChecking=no -o ConnectTimeout=30 -l rsync repo.monroe-system.eu rsync --server -mtrze.iLsfxC --timeout=30 --remove-source-files . /experiments/monroe
root       934  0.0  0.0  11228  3176 ?        S    09:26   0:00  |           \_ /bin/bash /usr/bin/monroe-rsync-results
root       992  0.0  0.0  13084  3380 ?        S    09:26   0:00  |               \_ rsync -r -t -z --exclude tstat/ --include */ --include *.json --include *.xz --include *.gz --exclude * --prune-empty-dirs --chmod=644 -e ssh -i /etc/keys/repo.monroe-system.eu -p 2280 -oStrictHostKeyChecking=no -o ConnectTimeout=30 --remove-source-files --timeout=30 /experiments/monroe/ rsync@repo.monroe-system.eu:/experiments/monroe
root       993  0.0  0.1  47512  6044 ?        S    09:26   0:00  |                   \_ ssh -i /etc/keys/repo.monroe-system.eu -p 2280 -oStrictHostKeyChecking=no -o ConnectTimeout=30 -l rsync repo.monroe-system.eu rsync --server -mtrze.iLsfxC --timeout=30 --remove-source-files . /experiments/monroe
root      8928  0.0  0.0  11200  3088 ?        S    09:35   0:00  |               \_ /bin/bash /usr/bin/monroe-rsync-results
root      9038  0.0  0.0  13084  3392 ?        S    09:35   0:00  |                   \_ rsync -r -t -z --exclude tstat/ --include */ --include *.json --include *.xz --include *.gz --exclude * --prune-empty-dirs --chmod=644 -e ssh -i /etc/keys/repo.monroe-system.eu -p 2280 -oStrictHostKeyChecking=no -o ConnectTimeout=30 --remove-source-files --timeout=30 /experiments/monroe/ rsync@repo.monroe-system.eu:/experiments/monroe
root      9039  0.0  0.1  47512  5872 ?        S    09:35   0:00  |                       \_ ssh -i /etc/keys/repo.monroe-system.eu -p 2280 -oStrictHostKeyChecking=no -o ConnectTimeout=30 -l rsync repo.monroe-system.eu rsync --server -mtrze.iLsfxC --timeout=30 --remove-source-files . /experiments/monroe
monroe   19363  0.0  0.0  12720   940 pts/0    S+   09:52   0:00              \_ grep -i rsync

We should have a check/semaphore, probably at L285 below, to check if we have a concurrent transfer already:

Utilities/monroe-experiments/usr/bin/monroe-experiments

Line 285 in 51e77d8

monroe-rsync-results || true

or somewhere deeper.