error TASK [libvirt : Create libvirt networks]

[asalkeld]

Describe the bug
I am trying to bring up a cluster and keep getting the below anisble error when runing "TASK [libvirt : Create libvirt networks]".

To Reproduce
Note: I have had an assisted-installer cluster on this machine before (but ran a "make destoy" in that repo first).

My config.sh has the following set
CI_TOKEN
PERSONAL_PULL_SECRET
OPENSHIFT_RELEASE_TYPE=ci

I ran

make clean all

observed behavior

TASK [libvirt : Create libvirt networks] ***********************************************************************************************************************************************************
task path: /opt/work/dev-scripts/metal3-dev-env/vm-setup/roles/libvirt/tasks/network_setup_tasks.yml:29
fatal: [localhost]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'template'. Error was a <class 'ansible.errors.AnsibleError'>, original message: template error while templating string: expected token '=', got '.'. String: {% set nat_port_range = item.nat_port_range|default([1024, 65535]) %}\n{% set netmask_v4 = item.netmask_v4|default(\"\") %}\n{% set prefix_v6 = item.prefix_v6|default(\"\") %}\n\n{% if item.dns.options is defined %}\n<network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'>\n  <dnsmasq:options>\n    <dnsmasq:option value='{{ item.dns.options }}'/>\n  </dnsmasq:options>\n{% else %}\n<network>\n{% endif %}\n\n  <name>{{ item.name }}</name>\n  <bridge name='{{ item.bridge }}'/>\n\n{% if item.forward_mode is defined %}\n  <forward mode='{{ item.forward_mode }}'>\n  {% if item.forward_mode == 'nat' %}\n    <nat>\n      <port start='{{ nat_port_range[0] }}' end='{{ nat_port_range[1] }}' />\n    </nat>\n  {% endif %}\n  </forward>\n{% endif %}\n\n{% if item.virtualport_type is defined %}\n      <virtualport type='{{ item.virtualport_type }}'/>\n{% endif %}\n\n{# IPv4 Configuration #}\n{% if item.address_v4 is defined and item.address_v4 != '' and item.forward_mode != 'bridge' %}\n  <ip address='{{ item.address_v4 }}' netmask='{{ netmask_v4 }}'>\n  {% if item.dhcp_range_v4 is defined %}\n    <dhcp>\n      <range start='{{ item.dhcp_range_v4[0] }}' end='{{ item.dhcp_range_v4[1] }}'/>\n    {% set ns = namespace(index=0) %}\n    {% for flavor in flavors %}\n      {% set numflavor = lookup('vars', 'num_' + flavor + 's')|default(0)|int %}\n      {% for num in range(0, numflavor) %}\n        {% set ironic_name = ironic_prefix + flavor + \"_\" + num|string %}\n        {% set hostname_format = lookup('vars', flavor + '_hostname_format', default=flavor + '-%d') %}\n        {% set hostname = hostname_format % num %}\n      <host mac='{{ node_mac_map.get(ironic_name).get(item.name)}}' name='{{hostname}}' ip='{{item.dhcp_range_v4[0]|ipmath(ns.index|int)}}'/>\n        {% set ns.index = ns.index + 1 %}\n      {% endfor %}\n    {% endfor %}\n    </dhcp>\n  {% endif %}\n  </ip>\n  {% if item.domain is defined %}\n  <domain name='{{ item.domain }}' localOnly='yes'/>\n  {% endif %}\n  {% if item.dns is defined %}\n  <dns>\n    {% for host in item.dns.hosts %}\n    <host ip='{{ host.ip }}'>\n      {% for name in host.hostnames %}\n      <hostname>{{ name }}</hostname>\n      {% endfor %}\n    </host>\n    {% endfor %}\n    {% if item.dns.srvs is defined %}\n      {% for srv in item.dns.srvs %}\n    <srv service='{{ srv.name }}' protocol='{{ srv.protocol }}' domain='{{ srv.domain }}' port='{{ srv.port }}' target='{{ srv.target }}' />\n      {% endfor %}\n    {% endif %}\n    {% if item.dns.forwarders is defined %}\n      {% for forwarder in item.dns.forwarders %}\n    <forwarder domain='{{ forwarder.domain }}' addr='{{ forwarder.addr }}' />\n      {% endfor %}\n    {% endif %}\n  </dns>\n  {% endif %}\n{% endif %}\n{# End IPv4 Configuration #}\n\n{# IPv6 Configuration #}\n{% if item.address_v6 is defined and item.address_v6 != '' and item.forward_mode != 'bridge' %}\n  <ip family=\"ipv6\" address='{{ item.address_v6 }}' prefix='{{ prefix_v6 }}'>\n  {% if item.dhcp_range_v6 is defined %}\n    <dhcp>\n      <range start='{{ item.dhcp_range_v6[0] }}' end='{{ item.dhcp_range_v6[1] }}'/>\n    {% set ns = namespace(index=0) %}\n    {% for flavor in flavors %}\n      {% set numflavor = lookup('vars', 'num_' + flavor + 's')|default(0)|int %}\n      {% for num in range(0, numflavor) %}\n        {% set ironic_name = ironic_prefix + flavor + \"_\" + num|string %}\n        {% set hostname_format = lookup('vars', flavor + '_hostname_format', default=flavor + '-%d') %}\n        {% set hostname = hostname_format % num %}\n        <host id='00:03:00:01:{{ node_mac_map.get(ironic_name).get(item.name)}}' name='{{hostname}}' ip='{{item.dhcp_range_v6[0]|ipmath(ns.index|int)}}'/>\n        {% set ns.index = ns.index + 1 %}\n      {% endfor %}\n    {% endfor %}\n    </dhcp>\n  {% endif %}\n  </ip>\n  {% if item.domain is defined %}\n  <domain name='{{ item.domain }}' localOnly='yes'/>\n  {% endif %}\n  {% if item.dns is defined %}\n  <dns>\n    {% for host in item.dns.hosts %}\n    <host ip='{{ host.ip }}'>\n      {% for name in host.hostnames %}\n      <hostname>{{ name }}</hostname>\n      {% endfor %}\n    </host>\n    {% endfor %}\n    {% if item.dns.srvs is defined %}\n      {% for srv in item.dns.srvs %}\n    <srv service='{{ srv.name }}' protocol='{{ srv.protocol }}' domain='{{ srv.domain }}' port='{{ srv.port }}' target='{{ srv.target }}' />\n      {% endfor %}\n    {% endif %}\n    {% if item.dns.forwarders is defined %}\n      {% for forwarder in item.dns.forwarders %}\n    <forwarder domain='{{ forwarder.domain }}' addr='{{ forwarder.addr }}' />\n      {% endfor %}\n    {% endif %}\n  </dns>\n  {% endif %}\n{% endif %}\n{# End IPv6 Configuration #}\n\n{% if item.portgroup is defined %}\n  {% for portgroup in item.portgroup %}\n  <portgroup name='{{ portgroup.name }}'>\n    {% if portgroup.vlan is defined %}\n    <vlan>\n      {% for vlan in portgroup.vlan %}\n      <tag id='{{ vlan.tag }}'/>\n      {% endfor %}\n    </vlan>\n    {% endif %}\n  </portgroup>\n  {% endfor %}\n{% endif %}\n\n</network>\n"
}

[hardys]

@asalkeld Hmm this normally happens when there's an error in the j2 template but I don't think anything changed in that area (the metal3-dev-env version we use is pinned ref https://github.com/openshift-metal3/dev-scripts/blob/master/01_install_requirements.sh#L22)

I wonder if this is related to the ansible version - what version are you using?

[asalkeld]

ansible --version
ansible 2.9.8
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/angus/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.8 (default, Dec 5 2019, 15:45:45) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

[hardys]

Ok not the ansible version, looks the same as my environment.

I suspect the assisted install changed the environment in some way that's altered the variables input to the j2 templating - probably the easiest way to debug is hack https://github.com/metal3-io/metal3-dev-env/blob/master/vm-setup/roles/libvirt/tasks/network_setup_tasks.yml#L34 to move the "template", "network.xml.j2" part into an explicit task as IIRC that yields a clearer error on failure than doing it inside the lookup.

If you can dump the vars consumed in the template as a task that may also show us what the difference is.

[hardys]

Ok having debugged in the environment provided by @asalkeld I discovered this is caused by an old version of python-jinja2:

$ sudo pip3 freeze | grep Jinja2
Jinja2==2.7.2
$ rpm -qa | grep -i jinja
python3-jinja2-2.10.1-2.el8_0.noarch

It seems something pip installed an old 2.7.2 version, which lacks namespace support (added in 2.10) so this part of the templating breaks:

https://github.com/metal3-io/metal3-dev-env/blob/master/vm-setup/roles/libvirt/templates/network.xml.j2#L37..L45

Unfortunately the error generated isn't super clear, and evidently we're lacking sufficient validation of the required version.

Looking at the logs it seems that ansible/jinja2 was already installed on the first run of 01_install_requirements.sh - any idea what would have installed that @asalkeld - could it have used a pinned j2 version?

[asalkeld]

I am not sure @hardys it must have been pulled in as a dep. I'll keep an eye on it.