Get the IDR playbooks working smoothly with multiple deployments

[manics]

Replaces vm_prefix with idr_environment in all openstack IDR playbooks, adds this as a group prefix in all IDR playbooks.

This means every server in an IDR deployment is in the following three groups

1. one of database-hosts omero-hosts gateway-hosts
2. {{idr_environment}}-hosts
3. one of {{idr_environment}} -database-hosts {{idr_environment}} -omero-hosts {{idr_environment}} -gateway-hosts

In practice this means it is possible to configure variables for

1. all servers by purpose (database, omero, gateway)
2. all servers in a given deployment (idr_environment)
3. a server by purpose in a given deployment (intersection of the above)

At present (3) should only contain one server (which can be addressed as e.g. {{idr_environment}} -database-hosts[0], if we implement load-balancing there may be more than 1.

This also reverts some changes to ansible.cfg from #112 in order to get this repo working, these changes should be reverted in a development or breaking branch until everyone's happy.

idr_environment defaults to idr so that the IDR playbooks can be run without any private configuration. In most cases you should set idr_environment to something else and add matching group vars files. For example see # IDR multinode omero in Vagrantfile

Fixes the following from the config issue:

• tty required for sudo (causes fatal: [demo3-omero]: FAILED! => {"failed": true, "msg": "Failed to get information on remote file (/etc/security/limits.d/95-omero.conf): MODULE FAILURE"} (idr-omero.yml))
• use of [0] for which host to use is dangerous since it assumes one type per cloud.

Includes:

• add packages needed for selinux setup #119 first commit only d25f4d1
• Fix Vagrantfile so vagrant up brings up all nodes #120
• Add optional cli utils #123
• Revert some performance improvement changes to ansible.cfg #125

For tidiness this PR should be rebased after merging the above three.

[snoopycrimecop]

Conflicting PR. Removed from build MANAGEMENT_TOOLS-merge#970. See the console output for more details.
Possible conflicts:

• PR Fix Vagrantfile so vagrant up brings up all nodes #120 manics 'Fix Vagrantfile so vagrant up brings up all nodes'
  • ansible/Vagrantfile

[snoopycrimecop]

Conflicting PR. Removed from build MANAGEMENT_TOOLS-merge#973. See the console output for more details.
Possible conflicts:

• PR Fix Vagrantfile so vagrant up brings up all nodes #120 manics 'Fix Vagrantfile so vagrant up brings up all nodes'
  • ansible/Vagrantfile

[joshmoore]

--exclude

This is the combined state of the other PRs. Getting those in first.

[manics]

As discussed I've refactored os-create.yml into a role, and added an example playbook.

[manics]

I've disabled the default nginx header in c135ef4, this shouldn't be needed any more since ome/openmicroscopy#4744 means the displayed host is generated on the client side not the server, and if it's enabled you always get redirected to idr-demo instead of whatever server you're trying to test..

[joshmoore]

Couple of questions:

• time to remove "exclude"?
• are the old top-level IDR playbooks slotted for removal?
• if we need to ship our own os_volume for a while happy to start bringing these into infrastructure (as with Feature #2731: added postgres import and dump ansible/ansible-modules-core#4614)

[manics]

time to remove "exclude"?

👍

are the old top-level IDR playbooks slotted for removal?

I'll remove them in a later PR once we know this works at EBI.

if we need to ship our own os_volume for a while happy to start bringing these into infrastructure (as with ansible/ansible-modules-core#4614)

The main benefit was for testing the changes to infrastructure, in production it should be a one off copy so the time penalty doesn't matter so much. If you get any timeout or "not ready" errors just re-run ansible-playbook after a few minutes.

[joshmoore]

This doesn't exist (yet?)

[manics]

True...that's in my other branch

[manics]

https://github.com/manics/infrastructure/tree/idr-terraform%2Bos-roles

[joshmoore]

For this to work, I have to remove clouds.yaml

[manics]

This is the public repo so the instructions are designed to work as far as possible without any private config. I assumed any internal instructions would be elsewhere

[joshmoore]

Understood, but in general, I think we may still need to re-evaluate openstack-private.py (later)

[manics]

Disabled omero_upgrade

[joshmoore]

Shooting for having this to be in 0.3.0-rc1 (of IDR)

[joshmoore]

No objections in chat. Only non-IDR change is the selinux blocker. Merging.