Michael Wadman

Viewing LibreNMS data in Grafana

Sun, 02 Jan 2022 00:00:00 +0000

Overview

LibreNMS is a great network device monitoring system, and Grafana is a wonderful tool for visualising and alerting on data from multiple sources.
Both are great examples of open-source software that you should be considering for your environment.

However, while LibreNMS does provide official integration with many external databases that you can then use as a data source in Grafana, if you don’t already have one of these in your environment it requires you to deploy these as well.
I’ve also personally run into performance issues with the Prometheus Pushgateway integration when pushing a large number of metrics, probably because Pushgateway isn’t designed for that use case.

Recently I stumbled upon a Reddit post by user /u/wjhinz which described a method for “(almost) natively” using LibreNMS as a data source in Grafana by using RRDReST as an API endpoint to access the RRD files that LibreNMS creates.
After testing this and confirming it works very well, I wanted to add to the Reddit post by documenting what a containerised deployment of this solution looks like.

Major kudos to both /u/tbotnz (for their work on RRDReST) and /u/wjhinz (for documenting this solution).

Containerised Deployment

LibreNMS

I’ll be using Docker Compose for this deployment, and starting with the LibreNMS docker example compose file as the base.
I’ve trimmed some services that aren’t needed (mstpd, syslog and SNMP traps), and included my compose file in an appendix section.

RRDReST

Next, we want to add an RRDReST container. As a container image doesn’t currently exist, we will need to create one.
Thanks to tbotnz, who has provided instructions on how to get RRDReST running in his Github repository, creating a container image is relatively easy.
Here’s the Dockerfile I’ve written:

FROM python:3.10-alpine
RUN apk add g++ make rrdtool
COPY requirements.txt /opt/RRDReST/requirements.txt
WORKDIR /opt/RRDReST
RUN pip3 install -r requirements.txt
COPY . /opt/RRDReST
ENTRYPOINT ["uvicorn", "rrdrest:rrd_rest"]
CMD ["--host", "0.0.0.0", "--port", "9000"]

I’ve built the container using this and pushed it to be pulled from Docker Hub as well.

To add this to our deployment, we simply need to add another service to the docker-compose.yml file and give the container access to the LibreNMS volume (so it can read the RRD files).
I’m using /opt/librenms as the mount point for the volume, as it mimics where data would be located in a non-containerised deployment.

services:
  ...
  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    volumes:
      - "./librenms:/opt/librenms"

As /u/wjhinz notes in their post, RRDReST needs read access to the RRD files, which should be fine with a docker-compose set up like this, but could be a good place to start investigations if it isn’t working for you.

Grafana

Finally, we want to add a Grafana container.

services:
  ...
  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    ports:
      - target: 3000
        published: 3000
        protocol: tcp
    environment:
      - "GF_INSTALL_PLUGINS=marcusolsson-json-datasource"

Note that I’m adding an environment variable for the installation of the JSON API plugin, as this is needed to interact with both the LibreNMS API as well as the RRDReST API.

docker-compose up

With a complete compose file, we can spin up the environment:

$ sudo docker-compose up -d
Creating network "librenms_grafana_default" with the default driver
Creating librenms            ... done
Creating grafana             ... done
Creating librenms_db         ... done
Creating librenms_dispatcher ... done
Creating librenms_redis      ... done
Creating librenms_memcached  ... done
Creating rrdrest             ... done

Adding data sources in Grafana

Creating LibreNMS user

To add LibreNMS as a data source in Grafana, we first need to create an API token in LibreNMS.
Visit the LibreNMS web interface and log in (in my case, “http://127.0.0.1:8000”, with the default credentials of librenms/librenms).

Because Grafana will only require read access, we’re going to create a new user with read-only access and then associate the token we create with that user.
Under ‘Settings -> Manage Users’, I created a user named “grafana”, changed the access level to “Global Read” and set a random password as we will only be using the API access.

Then, under ‘Settings -> API -> API Settings’, I created an API token for the grafana user:

Make sure to copy the token when creating it, as we’ll need that later when adding the data source into LibreNMS.

Adding LibreNMS data source in Grafana

After logging into the Grafana web interface (in my case, “http://127.0.0.1:3000”, with the default credentials of admin/admin), navigate to ‘Configuration -> Data Sources’ from the left pane.
Add a new data source and select the JSON API type from the list.

In the settings for the LibreNMS API data source, we need to change the URL and add the token authentication header.
The URL should be set to your LibreNMS host and port, with the API endpoint of “/api/v0” appended.
Then, under “Custom HTTP Headers”, select “Add Header”. The header name will be “X-Auth-Token” and the value will be the token string that you created above.
Save and test that the API endpoint is working for you, and you should get a green “Success” pop-up if everything is working.

My data source looked like the following when finished:

Adding RRDReST data source in Grafana

Like the above, we’re going to also add the RRDReST API as a new data source in Grafana too.
This will also be a JSON API, so select this from the data source type when creating it.

In the settings for the RRDReST API data source, we only need to change the URL.
Again, point this towards your RRDReST container host and port, and leave the endpoint blank.
Save and test that the API endpoint is working for you, and you should get a red “Unprocessable Entity” pop-up if everything is working. This error means the API is reachable, but the plugin can’t process the data returned (which is expected, because there isn’t any data returned, as we haven’t pointed it to an RRD file yet).

My data source looked like the following when finished:

Creating a dashboard

The last thing to do is to create a dashboard for the LibreNMS data to be visualised.
From the left pane in Grafana, select ‘Create -> Import’ and then upload the JSON file created by /u/wjhinz on Pastebin. I’ve also included it in an appendix section below in the case that the Pastebin link goes away for some reason.

After import, you’ll be prompted to name the dashboard, place it into a folder and select the data sources you created before. Here’s what mine looked like before import:

There was one last item that I needed to change to get data displayed on the graph panels, which was to set the timezone to UTC. I believe this is needed because LibreNMS stores (or RRDReST represents) data in UTC format, but Grafana queries it with the local browser timezone by default.
To do so, drop-down the time picker from the top right of the dashboard screen, select “Change Time Settings” and choose UTC.

If you’ve got devices in LibreNMS, they should appear up the top left of the page in a drop down menu along with a list of interfaces. After selecting a device and an interface, both the “Bandwidth In/Out” and “Errors In/Out” graphs will populate with information:

While the panels in the dashboard provided only show port throughput and errors, it’s pretty easy to pick up the other RRD files in a device directory and display those statistics as well.
For example, LibreNMS has created 64 other RRD files full of stats for my test device:

$ sudo docker exec -it rrdrest ls /opt/librenms/rrd/
Test_Device
$ sudo docker exec -it rrdrest ls /opt/librenms/rrd/Test_Device/ | grep -v port | wc -l
64

Conclusion

Providing RRDReST access to the RRD files that LibreNMS creates is not only easy, but it performs well too.
If you are already using Grafana in your environment to visualise, analyse and/or alert on data in a centralised fashion, then pulling SNMP metrics in through LibreNMS (and its’ device discovery feature) from devices that are otherwise unmonitored and using RRDReST as a data source is a great “quick win”.

Appendices

References

wjhinz’s Reddit Post
RRDReST Repository
RRDReST Container Image
Grafana JSON API Data source Plugin

Versions used

Desktop Machine: Ubuntu 20.04
Docker: 20.10.12
Docker Compose: 1.26.2
LibreNMS: 21.11.0
Grafana: 8.3.3

Full Code

Docker Compose file

version: "3.5"

services:
  db:
    image: mariadb:10.5
    container_name: librenms_db
    command:
      - "mysqld"
      - "--innodb-file-per-table=1"
      - "--lower-case-table-names=0"
      - "--character-set-server=utf8mb4"
      - "--collation-server=utf8mb4_unicode_ci"
    volumes:
      - "./db:/var/lib/mysql"
    environment:
      - "TZ=Pacific/Auckland"
      - "MYSQL_ALLOW_EMPTY_PASSWORD=yes"
      - "MYSQL_DATABASE=librenms"
      - "MYSQL_USER=librenms"
      - "MYSQL_PASSWORD=asupersecretpassword"
    restart: always

  memcached:
    image: memcached:alpine
    container_name: librenms_memcached
    environment:
      - "TZ=Pacific/Auckland"
    restart: always

  redis:
    image: redis:5.0-alpine
    container_name: librenms_redis
    environment:
      - "TZ=Pacific/Auckland"
    restart: always

  librenms:
    image: librenms/librenms:latest
    container_name: librenms
    hostname: librenms
    cap_add:
      - NET_ADMIN
      - NET_RAW
    ports:
      - target: 8000
        published: 8000
        protocol: tcp
    volumes:
      - "./librenms:/data"
    environment:
      - "TZ=Pacific/Auckland"
      - "PUID=1000"
      - "PGID=1000"
      - "DB_HOST=db"
      - "DB_NAME=librenms"
      - "DB_USER=librenms"
      - "DB_PASSWORD=asupersecretpassword"
      - "DB_TIMEOUT=60"
      - "REDIS_HOST=redis"
      - "REDIS_PORT=6379"
      - "REDIS_DB=0"
      - "MEMORY_LIMIT=256M"
      - "UPLOAD_MAX_SIZE=16M"
      - "OPCACHE_MEM_SIZE=128"
      - "REAL_IP_FROM=0.0.0.0/32"
      - "REAL_IP_HEADER=X-Forwarded-For"
      - "LOG_IP_VAR=remote_addr"
      - "MEMCACHED_HOST=memcached"
      - "MEMCACHED_PORT=11211"
      - "LIBRENMS_WEATHERMAP=false"
    restart: always

  dispatcher:
    image: librenms/librenms:latest
    container_name: librenms_dispatcher
    hostname: librenms-dispatcher
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - "./librenms:/data"
    environment:
      - "TZ=Pacific/Auckland"
      - "PUID=1000"
      - "PGID=1000"
      - "DB_HOST=db"
      - "DB_NAME=librenms"
      - "DB_USER=librenms"
      - "DB_PASSWORD=asupersecretpassword"
      - "DB_TIMEOUT=60"
      - "DISPATCHER_NODE_ID=dispatcher1"
      - "REDIS_HOST=redis"
      - "REDIS_PORT=6379"
      - "REDIS_DB=0"
      - "SIDECAR_DISPATCHER=1"
      - "MEMORY_LIMIT=256M"
      - "UPLOAD_MAX_SIZE=16M"
      - "OPCACHE_MEM_SIZE=128"
      - "REAL_IP_FROM=0.0.0.0/32"
      - "REAL_IP_HEADER=X-Forwarded-For"
      - "LOG_IP_VAR=remote_addr"
      - "MEMCACHED_HOST=memcached"
      - "MEMCACHED_PORT=11211"
      - "LIBRENMS_WEATHERMAP=false"
    restart: always

  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    volumes:
      - "./librenms:/opt/librenms"

  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    ports:
      - target: 3000
        published: 3000
        protocol: tcp
    environment:
      - "GF_INSTALL_PLUGINS=marcusolsson-json-datasource"

Grafana Dashboard JSON

{
  "__inputs": [
    {
      "name": "DS_RRDREST_API",
      "label": "RRDRest API",
      "description": "",
      "type": "datasource",
      "pluginId": "marcusolsson-json-datasource",
      "pluginName": "JSON API"
    },
    {
      "name": "DS_LIBRENMS_API",
      "label": "LibreNMS API",
      "description": "",
      "type": "datasource",
      "pluginId": "marcusolsson-json-datasource",
      "pluginName": "JSON API"
    }
  ],
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana"
    },
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph",
      "version": ""
    },
    {
      "type": "datasource",
      "id": "marcusolsson-json-datasource",
      "name": "JSON API",
      "version": "1.2.1"
    }
  ],
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": null,
  "iteration": 1626152089476,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_RRDREST_API}",
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "unit": "bps"
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 15,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.3.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "body": "",
          "cacheDurationSeconds": 0,
          "fields": [
            {
              "jsonPath": "$.data[*].time",
              "name": "time",
              "type": "time"
            },
            {
              "jsonPath": "$.data[*].inoctets",
              "name": "inoctets",
              "type": "number"
            },
            {
              "jsonPath": "$.data[*].outoctets",
              "name": "outoctets",
              "type": "number"
            }
          ],
          "hide": false,
          "method": "GET",
          "params": [
            [
              "",
              ""
            ]
          ],
          "queryParams": "",
          "refId": "A",
          "urlPath": "?rrd_path=/opt/librenms/rrd/$device/port-id$portid.rrd&epoch_start_time=${__from:date:seconds}&epoch_end_time=${__to:date:seconds}"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Bandwidth In/Out",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "transformations": [
        {
          "id": "calculateField",
          "options": {
            "alias": "inMbps",
            "binary": {
              "left": "inoctets",
              "operator": "*",
              "reducer": "sum",
              "right": "8"
            },
            "mode": "binary",
            "reduce": {
              "include": [
                "inoctets"
              ],
              "reducer": "lastNotNull"
            },
            "replaceFields": false
          }
        },
        {
          "id": "calculateField",
          "options": {
            "alias": "outMbps",
            "binary": {
              "left": "outoctets",
              "operator": "*",
              "reducer": "sum",
              "right": "8"
            },
            "mode": "binary",
            "reduce": {
              "reducer": "sum"
            },
            "replaceFields": false
          }
        },
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "names": [
                "time",
                "inMbps",
                "outMbps"
              ]
            }
          }
        }
      ],
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:1546",
          "format": "bps",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "$$hashKey": "object:1547",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_RRDREST_API}",
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "unit": "bps"
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 15,
        "w": 24,
        "x": 0,
        "y": 15
      },
      "hiddenSeries": false,
      "id": 3,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.3.5",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "body": "",
          "cacheDurationSeconds": 0,
          "fields": [
            {
              "jsonPath": "$.data[*].time",
              "name": "time",
              "type": "time"
            },
            {
              "jsonPath": "$.data[*].inerrors",
              "name": "inerrors",
              "type": "number"
            },
            {
              "jsonPath": "$.data[*].outerrors",
              "name": "outerrors",
              "type": "number"
            }
          ],
          "hide": false,
          "method": "GET",
          "params": [
            [
              "",
              ""
            ]
          ],
          "queryParams": "",
          "refId": "A",
          "urlPath": "?rrd_path=/opt/librenms/rrd/$device/port-id$portid.rrd&epoch_start_time=${__from:date:seconds}&epoch_end_time=${__to:date:seconds}"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Errors In/Out",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "transformations": [],
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:1546",
          "format": "bps",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "$$hashKey": "object:1547",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": "30s",
  "schemaVersion": 26,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_LIBRENMS_API}",
        "definition": "$.devices[*].hostname",
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "device",
        "options": [],
        "query": {
          "cacheDurationSeconds": 300,
          "fields": [
            {
              "jsonPath": "$.devices[*].hostname"
            }
          ],
          "method": "GET",
          "queryParams": "",
          "urlPath": "/devices"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_LIBRENMS_API}",
        "definition": "$.ports[*].port_id",
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Interface",
        "multi": false,
        "name": "portid",
        "options": [],
        "query": {
          "cacheDurationSeconds": 300,
          "experimentalVariableTextField": "ifName",
          "experimentalVariableValueField": "id",
          "fields": [
            {
              "jsonPath": "$.ports[*].port_id",
              "name": "id"
            },
            {
              "jsonPath": "$.ports[*].ifName",
              "name": ""
            }
          ],
          "method": "GET",
          "queryParams": "",
          "urlPath": "/devices/$device/ports?columns=ifName%2Cport_id"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-12h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "utc",
  "title": "LibreNMS RRDReST",
  "uid": "mFveL2mm1",
  "version": 28
}

Resolving my DNS issues - Disabling systemd-resolved on Ubuntu 18.04

Sun, 22 Sep 2019 00:00:00 +0000

Overview

The below snippet from my ~/.bash_aliases should illustrate my feelings towards systemd-resolved:

$ grep resolved ~/.bash_aliases
alias fuck-resolved='sudo systemctl restart systemd-resolved'

For the longest time, I’ve put up with the pain-in-my-ass that is systemd-resolved and the constant issues that it has given me (not to mention the time spent troubleshooting various issues). Today, I intend to resolve my issues once and for all.

There are a ton of “guides” already on the internet (a google search for “disable systemd-resolved” has 179,000 results at the time of writing this).
The blog post My War on Systemd-resolved is a good, quick read and gives some ideas on why the google search results differ ever-so-slightly from each other (but please don’t follow his proposed solution).
Like the above post, I intend to shut down systemd-resolved permanently, and document how (for Ubuntu 18.04 at least).

Disabling systemd-resolved

First up is disabling the systemd-resolved service.
Easy enough using systemctl:

$ sudo systemctl disable systemd-resolved
$ sudo systemctl stop systemd-resolved

Next, we’ll remove the existing “/etc/resolv.conf” file, which is currently a symbolic link to “/run/systemd/resolve/stub-resolv.conf”:

$ ls /etc/resolv.conf -al
lrwxrwxrwx 1 root root 39 Jun 16  2018 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
$ sudo rm /etc/resolv.conf

Reconfiguring DNS

There are now two paths you can take to configure your DNS, depending on whether you receive your DNS configuration via DHCP or not.
If you do, then we’ll configure NetworkManager to look after the “/etc/resolv.conf” file.
If you don’t, then simply configure this file statically yourself and you’re good to go.

Using NetworkManager

If you want to use the settings given out via DHCP, we need to reconfigure NetworkManager so that it starts to set this for us.
This is as simple as editing the file “/etc/NetworkManager/NetworkManager.conf” and setting (or adding, if not already present) the dns and rc-manager keys under the [main] section, as below:

[main]
dns=default
rc-manager=resolvconf

Then we’ll restart NetworkManager so that it’ll create us a new “/etc/resolv.conf” file:

sudo systemctl restart network-manager

We can now verify that the new configuration has been picked up by checking the contents of “/etc/resolv.conf”:

$ cat /etc/resolv.conf
# Generated by NetworkManager
search example.com
nameserver 192.168.0.1
nameserver 192.168.0.2

Manual configuration

If you’re not receiving DNS settings over DHCP, or you want to have manual control over this, then you can skip the NetworkManager configuration and simply start manually configuring the “/etc/resolv.conf” file yourself.

Appendices

References

My War on Systemd-resolved
Disabling systemd-resolvd service on Ubuntu systems
AskUbuntu - How do I disable systemd-resolved and replace with something sane on Ubuntu 18?
How to correctly disable systemd-resolved on Ubuntu 18.04

Versions used

Desktop Machine: kubuntu-18.04.3

OpenStack Lab Network - Deployment Preparation

Sun, 14 Apr 2019 00:00:00 +0000

Overview

This post is the tenth in a series that plans to document my progress through installing and configuring a small OpenStack Lab.

For other posts in this series, see the overview section of the introduction post.

In the last post, we installed and configured the FRR suite on our OpenStack hosts. In this post, we’ll start making the final touches before we run OpenStack-Ansible.

Configuration

This post will be a quick one, with the goal of preparating our OpenStack hosts for the OpenStack services to be deployed to them.

The steps that I’m going to cover will be a direct translation of what is listed in OpenStack-Ansible’s deployment guide. Specifically, the following pages:

Let’s start with the deployment host preparation.

Deployment Host

The deployment guide recommends that (in a testing environment) one of the control nodes act as the deployment host as well.
In production this role would instead be handed to a host that won’t be participating in the OpenStack services, to avoid potential conflicts occurring during initial deployment or upgrades (as the deployment host will act as the source of any future updates to OpenStack services).

Ansible Playbook and Role

With that in mind, let’s create a new role and add this to our existing playbook.

---
- name: Configuring OpenStack hosts
  hosts: openstack_hosts
  become: true
  gather_facts: true
  roles:
    - role: openstack-interfaces
      tags:
        - openstack-interfaces
    - role: openstack-routing
      tags:
        - openstack-routing

- name: Prepare deployment host
  hosts: openstack-control
  become: true
  gather_facts: true
  roles:
    - role: openstack-deployment
      tags:
        - openstack-deployment

Note that instead of adding this role onto the existing “Configuring OpenStack hosts” entry, I’ve defined another entry.
By doing this, I can specify that the role is only to be run on hosts: openstack-control.

We’ll create the skeleton of our new role too.

$ mkdir -p /etc/ansible/roles/openstack-deployment/tasks/

Packages

As always, our first step is to ensure the required packages are installed on our host.
The deployment guide gives us a list of required packages.

Here’s what the first task in our file “/etc/ansible/roles/openstack-interfaces/tasks/main.yml” looks like:

- name: Install prerequisite packages
  apt:
    name:
      - aptitude
      - build-essential
      - git
      - ntp
      - ntpdate
      - openssh-server
      - python-dev
      - sudo
    state: latest
    cache_valid_time: 3600

Clone Repository

Next, the guide says to clone their repository into the local directory “/opt/openstack-ansible”.
For this, we’ll use Ansible’s “git” module.

- name: Clone openstack-ansible repository
  git:
    repo: "https://github.com/openstack/openstack-ansible"
    dest: "{{ openstack_ansible_exec_dir }}"
    version: "stable/stein"

This module has a ton of parameters that are able to be passed in, but we’re going to stay simple and use just the following:

repo - The address to clone the repository from. I’ll use the projects Github mirror as I’ve had issues with the official repository link in the past.
dest - The directory to clone the repository into.
version - The version of the repository to clone. In our case, we’re going to reference the branch named “stable/stein”, as we’ll be installing the OpenStack version named “Stein” using OpenStack-Ansible.

I’ve also referenced the variable “openstack_ansible_exec_dir”, which isn’t yet defined.
I’m going to define this variable (as “/opt/openstack-ansible”), along with another variable we’ll use later, in the defaults file (“defaults/main.yml”) for this role:

---
openstack_ansible_exec_dir: "/opt/openstack-ansible"
openstack_ansible_config_dir: "/etc/openstack_deploy"

SSH Authentication

During the running of the OpenStack-Ansible playbooks, Ansible on the deployment host will attempt to connect as the root user to other remote nodes.
In order for this to work, we’ll need to create and then set the root users’ public key on the deployment (control) host as an authorised key on our compute host.

- name: Create a SSH key for the root user
  command: ssh-keygen -q -t rsa -b 4096 -f /root/.ssh/id_rsa -C "" -N ""
  args:
    creates: /root/.ssh/id_rsa.pub

The above command will create the ssh-key if it doesn’t already exist.
In the future, we should be able to use the new “openssh_keypair” module, but this is currently only available in the development release.

To store the contents of the public key file as a variable, we’ll use Ansible’s “slurp” and “set_fact” modules.

- name: Get root user public key
  slurp:
    path: /root/.ssh/id_rsa.pub
  register: root_pub_key

- set_fact:
    openstack_deploy_pub_key: "{{ root_pub_key['content'] | b64decode }}"

“slurp” will gather the contents of a file on a remote host. Together with the “register” keyword, we can save this as a (base-64 encoded) variable.

We then use “set_fact” to decode and save the result as another variable.
We need to save this off again, as the “register” action only saves the variable on the host the playbook is running on at the time, in our case the OpenStack control host, and we want this available across all of our hosts.

Bootstrap OpenStack-Ansible

The last step for the deployment host is to run the bootstrap script that was cloned in the step before.

- name: Run bootstrap-ansible.sh script
  command: ./scripts/bootstrap-ansible.sh
  args:
    chdir: "{{ openstack_ansible_exec_dir }}"
    creates: "/usr/local/bin/openstack-ansible"

The creates argument is a way of telling the task whether it needs to be run or not, by saying what file is created if it has been run successfully.
I’ve used the file “/usr/local/bin/openstack-ansible” as this is the last (we want the creation of the file to be as near to the end of the script as possible) file that is always created when the script is run.

Target Hosts

Onto the target hosts. The guide for which is the next page on from the deployment host guide.

Ansible Playbook and Role

We’ll add our new role to the playbook first:

---
- name: Configuring OpenStack hosts
  hosts: openstack_hosts
  become: true
  gather_facts: true
  roles:
    - role: openstack-interfaces
      tags:
        - openstack-interfaces
    - role: openstack-routing
      tags:
        - openstack-routing

- name: Prepare deployment host
  hosts: openstack-control
  become: true
  gather_facts: true
  roles:
    - role: openstack-deployment
      tags:
        - openstack-deployment

- name: Configuring OpenStack hosts
  hosts: openstack_hosts
  become: true
  gather_facts: true
  roles:
    - role: openstack-preparation
      tags:
        - openstack-preparation

And our role creation.

$ mkdir -p /etc/ansible/roles/openstack-preparation/tasks/

Packages

You know the drill by now. Pre-requisitie packages first.

- name: Install prerequisite packages
  apt:
    name:
      - bridge-utils
      - debootstrap
      - ifenslave
      - ifenslave-2.6
      - lsof
      - lvm2
      - openssh-server
      - python
      - sudo
      - tcpdump
      - vlan
    state: latest
    cache_valid_time: 3600

Kernel Modules

After installing packages, the guide then says to complete the following:

Install the “linux-image-extra” package. For our case, this would be “linux-image-extra-virtual-hwe-18.04”, which doesn’t install anything other than a copyright text file in “/usr/share/doc”, at least at the time of writing this.
Configure the chrony service. The astute of you will notice that I have omitted the chrony package from the list in the previous step. This is because we really don’t need to worry about time drift in an ephemeral lab deployment like ours. Even in a production environment, I would consider just configuring ntpd correctly instead.
Load the kernel modules “8021q” and “bonding”. Since we aren’t bonding interfaces in our topology, we’ll just enable the 8021q module below.

- name: Enable required kernel modules
  modprobe:
    name: 8021q

Yes, that simple.

SSH Authentication

Carrying on from where we left off in the SSH Authentication section for the deployment host, we now have a variable which stores the public key of the root user.
All we need to do now is set this as an authorised key on all other hosts. Luckily there is an Ansible module for doing just that.

- name: Set root user authorised ssh key
  authorized_key:
    user: root
    key: "{{ hostvars['openstack-control']['openstack_deploy_pub_key'] }}"

Storage

Because we’ve already set up the networking, the last step is to set up LVM storage for the Cinder (block storage) service and lxc.
We’ll be utilising the second hard disk that we set up on both of our OpenStack hosts in the Vagrant configuration.

- name: Create required partitions
  parted:
    name: "{{ item.name }}"
    state: present
    device: /dev/sdb
    number: "{{ item.number }}"
    part_start: "{{ item.part_start }}"
    part_end: "{{ item.part_end }}"
    label: gpt
    flags: [ lvm ]
  loop:
    - name: cinder-volumes
      number: 1
      part_start: '0%'
      part_end: '50%'
    - name: lxc
      number: 2
      part_start: '50%'
      part_end: '100%'

- name: Create required LVM volume groups
  lvg:
    vg: "{{ item.vg }}"
    pvs: "{{ item.pvs }}"
    vg_options: --metadatasize 2048
  loop:
    - vg: cinder-volumes
      pvs: /dev/sdb1
    - vg: lxc
      pvs: /dev/sdb2

First up we use the “parted” module to split our disk into two equal sized partitions. The naming of these is important, as the OpenStack-Ansible playbooks look for these specific names when running.
Next, we create LVM volume groups using the “lvg” module. Note that we don’t need to create the physical volumes separately, as these are created automatically by the module if they don’t exist.

Conclusion

In this post, we’ve completed the final preparation steps (a bit of an oxymoron, I know) required on our OpenStack hosts.

Appendices

References

Ansible - “apt” module
Ansible - “git” module
Ansible - “command” module
Ansible - “slurp” module
Ansible - “set_fact” module
Ansible - “authorized_key” module
Ansible - “parted” module
Ansible - “lvg” module
OpenStack-Ansible - Prepare the deployment host
OpenStack-Ansible - Prepare the target hosts

Versions used

Desktop Machine: kubuntu-18.04
VirtualBox: virtualbox-5.2.18
Vagrant: 2.2.4
Cumulus VX Vagrant Box: CumulusCommunity/cumulus-vx (virtualbox, 3.7.5)
Ubuntu Server Vagrant Box: geerlingguy/ubuntu1804 (virtualbox, 1.0.7)
Ansible: 2.7.10

OpenStack Lab Network - Host Routing

Sun, 03 Feb 2019 00:00:00 +0000

Overview

This post is the ninth in a series that plans to document my progress through installing and configuring a small OpenStack Lab.

For other posts in this series, see the overview section of the introduction post.

In the last post, we configured the network interfaces on our OpenStack hosts. In this post, we’ll cover the installation and configuration of our routing protocol suite, Free Range Routing (FRR), on our OpenStack hosts.

Configuration

As I’ve already gone over what FRR is and how it is configured, I’m not going to cover that in too much detail in this post.
What I will cover is the differences between FRR on our switches and our OpenStack hosts, mainly the installation of FRR as this is not already present on our hosts.

At the end of this post, we’ll have:

OSPF neighborships from the hosts to their connected leaf switches.
BGP peerings from the hosts to the spine switches/route reflectors, with the EVPN family/capability negotiated.

Ansible Playbook and Role

Let’s start with our playbook.

---
- name: Configuring OpenStack hosts
  hosts: openstack_hosts
  become: true
  gather_facts: true
  roles:
    - role: openstack-interfaces
      tags:
        - openstack-interfaces
    - role: openstack-routing
      tags:
        - openstack-routing

And our role creation.

$ mkdir -p /etc/ansible/roles/openstack-routing/tasks/

FRR Prerequisites

FRR Packages

Because FRR does not come preinstalled on our hosts, our first tasks will cover the installation of the required packages.
This step is made a little trickier for us because the latest versions of FRR are not present in any public repository (not even their own one).
Instead, the installation (and any upgrade) of FRR will need to be done by manually downloading the packages from their releases page on Github (or directly from the projects CI site, by navigating to the “artifacts” page for the version you want to install - Ubuntu 18.04 x86_64, as an example).

Side note: If you’re not looking for the features from the newest release, then Cumulus do provide a public repository for version 4.0. My recommendation would be to avoid this if at all possible, as this doesn’t seem to be updated frequently.
Instructions on how to use their repository can be found on their website here.

Luckily for us, some people have already created a role that includes tasks for the installation of FRR (and supports Ubuntu 18.04 hosts like ours) - ansible-frr.
However, the role they’ve written also includes tasks for steps that we either want to complete ourselves (configuration of FRR) or don’t want to include at all (Installation of their custom scripts for monitoring), so instead of calling the entire role in our playbook we’re going to “include” this role in our tasks file and ask that only the installation tasks are completed.

The first thing we need to do is install the ansible-frr role as a git submodule so that we can reference it:

/etc/ansible$ git submodule add https://github.com/mrlesmithjr/ansible-frr.git roles/ansible-frr

After which, we can use Ansible’s include_role module with the argument tasks_from to specify that we only want to run the tasks in the “debian.yml” tasks file (the ones needed to install FRR on Ubuntu).
This is what our “tasks/main.yml” looks like:

- name: Install FRR
  include_role:
    name: ansible-frr
    tasks_from: debian

By default, the above tasks file will download and install the latest stable version of FRR on our hosts.
Thanks, ansible-frr!

Kernel Version

Before we move on to the configuration of FRR, there’s one other prerequisite that the above role doesn’t consider - the kernel version of the host.
The version being used on our OpenStack hosts becomes important when we take a look at the minimum kernel required for specific features of FRR, as outlined on either on FRR’s Github Wiki or on their official documentation site.

According to these pages, to be able to run EVPN (with full feature support/at full performance) on our Linux based Ubuntu hosts we need to be running a minimum kernel version of 4.18.
Unfortunately, if we look at our Ubuntu hosts we see that these are installed with kernel 4.15:

$ uname -r
4.15.0-29-generic

Luckily, Ubuntu 18.04 has what is called a “hardware enablement kernel”, that we can install via a package from the default repositories, which will bring us up to the desired 4.18 kernel version.

To install this using Ansible, I am first going to write a task that will check our hosts to see if they need to be upgraded, and place this before our “Install FRR” task:

---
- name: Check if minimum linux kernel version is met
  include_tasks: kernel-upgrade.yml
  when: ansible_kernel is version (openstack_routing_kernel_minimum, '<')

- name: Install FRR
  include_role:
    name: ansible-frr
    tasks_from: debian

Similar to the above use of the include_role module, we’re going to use the module include_tasks to run the tasks inside of another file (“kernel-upgrade.yml”) that we’ll create next.

Also note the use of the ansible fact “ansible_kernel”, which is really helpful with this task. The when clause of the action takes that fact and checks whether it is less than the variable “openstack_routing_kernel_minimum” to see if the included tasks need to be run.
We haven’t yet defined the variable “openstack_routing_kernel_minimum” though, so let’s do that before we move on:

In defaults/main.yml:

---
openstack_routing_kernel_minimum: 4.18

Finally, this is what the “tasks/kernel-upgrade.yml” file ends up looking like:

---
- name: Upgrade to required linux kernel version
  apt:
    name: linux-generic-hwe-18.04
    state: present
    cache_valid_time: 3600

- name: Reboot host after kernel upgrade
  reboot:

This is also pretty simple. Install the hardware enablement kernel package and then use the reboot module afterwards so that the new kernel is used (installing the package enables the kernel on the next boot).

We’re done with our prerequisites. Onto configuration.

FRR Configuration

Because I’ve covered the major points of FRR, OSPF and BGP in a previous post from the switch point of view, I’ll try and make the next sections brief and again only cover the differences.

Kernel Parameters

One of those differences is that our Ubuntu hosts are not readily built to act as “routers”, due to the default settings of the kernel in Ubuntu 18.04 server.

One good example of this is the switch “net.ipv4.ip_forward”, which by default is set to “0” (or “false” in boolean terms).
With this setting left as is, the host will never forward packets that it receives either from other external devices (switches) or from internal virtual devices (virtual machines). This is pretty bad if we want virtual machines on our two hosts to be able to talk to each other.

Not to worry, these settings are quite easy to change.
We’ll use the trusty template module, along with a handler to load any changes made.

First, our task:

- name: Create sysctl file for routing tweaks
  template:
    src: sysctl.j2
    dest: "{{ openstack_routing_sysctl_file }}"
  notify:
    - Load sysctl changes

Then we need to define the variable “openstack_routing_sysctl_file”.
I’ll do so in “defaults/main.yml”:

openstack_routing_sysctl_file: "/etc/sysctl.d/99free_range_routing.conf"

Our template file (I won’t cover every setting. If you’re interested, have a google!):

# 
# /etc/sysctl.d/99free_range_routing.conf

# Enables Routing
net.ipv4.ip_forward=1

# Routing Options
net.ipv4.conf.all.ignore_routes_with_linkdown=1

# Enables Unnumbered BGP/OSPF
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.lo.rp_filter=0
net.ipv4.conf.all.forwarding=1
net.ipv4.conf.default.forwarding=1
net.ipv4.conf.default.arp_announce=2
net.ipv4.conf.default.arp_notify=1
net.ipv4.conf.default.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.all.arp_notify=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.icmp_errors_use_inbound_ifaddr=1

# ARP/NDP Garbage Collection
net.ipv4.neigh.default.gc_thresh2=7168
net.ipv4.neigh.default.gc_thresh3=8192
net.ipv4.neigh.default.base_reachable_time_ms=14400000

# Use neighborship info on selection of nexthop address for multipath hops
net.ipv4.fib_multipath_use_neigh=1

# Allows applications to work with VRF
net.ipv4.tcp_l3mdev_accept=1

Last, our “handlers/main.yml” file:

- name: Load sysctl changes
  command: "sysctl -p "

Base Configuration

Just like our switches, we need to tell FRR which of its’ daemons to load for the host to be active in our topology.
A template will do the trick here too:

- name: Set FRR daemons file
  template:
    src: "daemons.j2"
    dest: "/etc/frr/daemons"
  notify: Restart FRR

With our template looking like the below:

# {{ ansible_managed }}
# This file tells the frr package which daemons to start.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activation a daemon at the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= thendough. Check /etc/pam.d/frr, too.
#
# The watchfrr and zebra daemons are always started.
#
bgpd={{   "yes" if openstack_hosts_routing_bgp_enabled   else "no" }}
ospfd={{  "yes" if openstack_hosts_routing_ospf_enabled  else "no" }}
ospf6d={{ "yes" if openstack_hosts_routing_ospf6_enabled else "no" }}
ripd={{   "yes" if openstack_hosts_routing_rip_enabled   else "no" }}
ripngd={{ "yes" if openstack_hosts_routing_ripng_enabled else "no" }}
isisd={{  "yes" if openstack_hosts_routing_isis_enabled  else "no" }}
pimd={{   "yes" if openstack_hosts_routing_pim_enabled   else "no" }}
ldpd={{   "yes" if openstack_hosts_routing_ldp_enabled   else "no" }}
nhrpd={{  "yes" if openstack_hosts_routing_nhrp_enabled  else "no" }}
eigrpd={{ "yes" if openstack_hosts_routing_eigrp_enabled else "no" }}
babeld={{ "yes" if openstack_hosts_routing_babel_enabled else "no" }}
sharpd={{ "yes" if openstack_hosts_routing_sharp_enabled else "no" }}
pbrd={{   "yes" if openstack_hosts_routing_pbr_enabled   else "no" }}
bfdd={{   "yes" if openstack_hosts_routing_bfd_enabled   else "no" }}

#
# If this option is set the /etc/init.d/frr script automatically loads
# the config via "vtysh -b" when the servers are started.
# Check /etc/pam.d/frr if you intend to use "vtysh"!
#
vtysh_enable=yes
zebra_options="  --daemon -A 127.0.0.1 -s 90000000"
bgpd_options="   --daemon -A 127.0.0.1"
ospfd_options="  --daemon -A 127.0.0.1"
ospf6d_options=" --daemon -A ::1"
ripd_options="   --daemon -A 127.0.0.1"
ripngd_options=" --daemon -A ::1"
isisd_options="  --daemon -A 127.0.0.1"
pimd_options="   --daemon -A 127.0.0.1"
ldpd_options="   --daemon -A 127.0.0.1"
nhrpd_options="  --daemon -A 127.0.0.1"
eigrpd_options=" --daemon -A 127.0.0.1"
babeld_options=" --daemon -A 127.0.0.1"
sharpd_options=" --daemon -A 127.0.0.1"
pbrd_options="   --daemon -A 127.0.0.1"
staticd_options="--daemon -A 127.0.0.1"
bfdd_options="   --daemon -A 127.0.0.1"

# The list of daemons to watch is automatically generated by the init script.
watchfrr_options="-r '/usr/lib/frr/watchfrr.sh restart %s' -s '/usr/lib/frr/watchfrr.sh start %s' -k '/usr/lib/frr/watchfrr.sh stop %s'"

# for debugging purposes, you can specify a "wrap" command to start instead
# of starting the daemon directly, e.g. to use valgrind on ospfd:
#   ospfd_wrap="/usr/bin/valgrind"
# or you can use "all_wrap" for all daemons, e.g. to use perf record:
#   all_wrap="/usr/bin/perf record --call-graph -"
# the normal daemon command is added to this at the end.

There are two differences in the above file, as compared to our switches “daemons” file:

On our hosts, we do not need to specify that the “zebra” daemon is to run, as this is presumed by default in version 6.0.2 (the latest version at the time of writing).
We need to include the options that our daemons will be started (“$DAEMON_options=”). I’ve just used the defaults.

Before we dive into our routing protocol configuration, let’s also get our “frr.conf” file started.
The task:

- name: Set FRR running configuration
  template:
    src: "frr.conf.j2"
    dest: "/etc/frr/frr.conf"
  notify: Reload FRR

The handler:

- name: Reload FRR
  service:
    name: frr
    state: reloaded

And the template “frr.conf.j2”:

! {{ ansible_managed }}
!
frr defaults datacenter
!
hostname {{ inventory_hostname }}
!
service integrated-vtysh-config
!
log syslog informational
!
line vty
!

Nothing has changed here from our switches initial FRR configuration.

We’ll also add a task at the end of our role to start FRR after the configuration has been set:

- name: Start and Enable FRR
  service:
    name: frr
    state: started
    enabled: yes

OSPF

Seeing as our goal with OSPF is very close to that of our switches, with the only addition being that we’re advertising the bridge IP addressing, it’ll be no surprise that our configuration is very close on each as well:

!
router-id {{ openstack_host_loopback_address | ipaddr('address') }}
!
!
{% if openstack_hosts_routing_ospf_enabled == true %}
{# Configure switchports as OSPF point-to-point #}
{% for port, mac in openstack_hosts_network_interfaces.items() | sort %}
interface {{ port }}
  ip ospf network point-to-point
!
{% endfor %}
!
{% for bridge, address in openstack_host_network_bridges.items() | sort %}
{% if address is not none %}
interface {{ bridge }}
  no link-detect
!
{% endif %}
{% endfor %}
{% set host_number = (openstack_host_loopback_address | ipaddr('address')).split('.')[3] %}
!
router ospf
  ospf router-id {{ openstack_host_loopback_address | ipaddr('address') }}
  network {{ openstack_host_loopback_address }} area 0
  passive-interface lo
{% for bridge, address in openstack_host_network_bridges.items() | sort %}
{% if address is not none %}
  network {{ address | ipaddr('network/prefix') }} area {{ host_number }}
  passive-interface {{ bridge }}
{% endif %}
{% endfor %}
  area {{ host_number }} range 10.{{ host_number }}.0.0/16
{% endif %}
!

I’ll cover some sections of note.

{% for bridge, address in openstack_host_network_bridges.items() | sort %}
{% if address is not none %}
interface {{ bridge }}
  no link-detect
!
{% endif %}
{% endfor %}

FRR considers the bridge interfaces we’ve configured as ‘down’, because don’t have any member interfaces (yet).
To bypass this, we need tell FRR to not care (detect) whether the link is up or down when deciding whether to advertise the associated networks.

{% set host_number = (openstack_host_loopback_address | ipaddr('address')).split('.')[3] %}
...
  area {{ host_number }} range 10.{{ host_number }}.0.0/16

Because we’ve addressed our bridge interfaces in such a way that they can be summarised on each host, we can use the area x range 10.x.0.0/16 command.
This will tell the host to send out a summary route (/16) instead of individual routes.
To determine how to summarise, we take the take octet of the loopback address and set this to a variable using Jinja2’s set command.

BGP

Our BGP is going to be very similar too, with the only changes being:

We don’t require logic to check if we need BGP because it’ll be present on all (well, “both” in our lab) hosts.
We don’t need the route reflector configuration, because we’re configuring our hosts as clients only.

!
{% if openstack_hosts_routing_bgp_enabled == true %}
router bgp {{ ibgp_autonomous_system }}
  bgp router-id {{ openstack_host_loopback_address | ipaddr('address') }}
{# Defines a peer group where common configuration to be defined for all peers #}
  neighbor iBGP-RRs peer-group
  neighbor iBGP-RRs remote-as internal
{# Forms a neighborship with spine switches/route reflectors #}
{% for spine in groups['openstack_cumulus_spines'] %}
  neighbor {{ hostvars[spine].cumulus_host_loopback_address | ipaddr('address') }} peer-group iBGP-RRs
{% endfor %}
{% if openstack_hosts_routing_bgp_evpn_enabled == true %}
{# Enables advetisement of EVPN address family #}
  address-family l2vpn evpn
    neighbor iBGP-RRs activate
    advertise-all-vni
  exit-address-family
{% endif %}
{% endif %}
!

Testing

After another vagrant up, the above configuration is applied and we can log into our hosts with vagrant ssh to check everything is working.
The below is taken from the perspective of one of the hosts.

The first check we should complete is to ensure FRR and its’ daemons are running:

$ sudo systemctl status frr.service
● frr.service - FRRouting
   Loaded: loaded (/lib/systemd/system/frr.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-02-17 03:13:41 UTC; 1h 8min ago
     Docs: https://frrouting.readthedocs.io/en/latest/setup.html
  Process: 570 ExecStart=/usr/lib/frr/frrinit.sh start (code=exited, status=0/SUCCESS)
    Tasks: 11 (limit: 4915)
   CGroup: /system.slice/frr.service
           ├─603 /usr/lib/frr/watchfrr -d -r /usr/lib/frr/watchfrr.sh restart %s -s /usr/lib/frr/watchfrr.sh start %s -k /usr/lib/frr/watchfrr.sh stop %s zebra bgpd ospfd staticd
           ├─631 /usr/lib/frr/zebra -d --daemon -A 127.0.0.1 -s 90000000
           ├─710 /usr/lib/frr/bgpd -d --daemon -A 127.0.0.1
           ├─841 /usr/lib/frr/ospfd -d --daemon -A 127.0.0.1
           └─968 /usr/lib/frr/staticd -d --daemon -A 127.0.0.1

Feb 17 03:13:41 vagrant watchfrr[603]: bgpd state -> up : connect succeeded
Feb 17 03:13:41 vagrant watchfrr[603]: ospfd state -> up : connect succeeded
Feb 17 03:13:41 vagrant watchfrr[603]: staticd state -> up : connect succeeded
Feb 17 03:13:41 vagrant watchfrr[603]: all daemons up, doing startup-complete notify

Looking good. Now let’s confirm our routing protocols look happy.

OSPF Testing

We’ll start with OSPF because that’s our routing underlay (BGP won’t come up without OSPF working first).

Let’s check neighborship to the leaf switches:

$ sudo vtysh

Hello, this is FRRouting (version 6.0.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

vagrant# show ip ospf neighbor

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
192.168.11.113    1 Full/DROther      37.310s 192.168.11.113  enp0s8:192.168.11.132     0     0     0
192.168.11.114    1 Full/DROther      37.518s 192.168.11.114  enp0s9:192.168.11.132     0     0     0

Are we receiving OSPF routes?

vagrant# show ip route ospf
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

O>* 192.168.11.101/32 [110/200] via 192.168.11.113, enp0s8 onlink, 00:09:45
  *                             via 192.168.11.114, enp0s9 onlink, 00:09:45
O>* 192.168.11.102/32 [110/200] via 192.168.11.113, enp0s8 onlink, 00:09:45
  *                             via 192.168.11.114, enp0s9 onlink, 00:09:45
O>* 192.168.11.111/32 [110/300] via 192.168.11.113, enp0s8 onlink, 00:09:45
  *                             via 192.168.11.114, enp0s9 onlink, 00:09:45
O>* 192.168.11.112/32 [110/300] via 192.168.11.113, enp0s8 onlink, 00:09:45
  *                             via 192.168.11.114, enp0s9 onlink, 00:09:45
O>* 192.168.11.113/32 [110/100] via 192.168.11.113, enp0s8 onlink, 00:09:45
O>* 192.168.11.114/32 [110/100] via 192.168.11.114, enp0s9 onlink, 00:09:45
O>* 192.168.11.131/32 [110/400] via 192.168.11.113, enp0s8 onlink, 00:09:46
  *                             via 192.168.11.114, enp0s9 onlink, 00:09:46
O   192.168.11.132/32 [110/0] is directly connected, lo, 01:13:49

This is great. Not only can we see that we’re receiving routes all the way from the other OpenStack host (192.168.11.131/32), but all of the routes are installed as ECMP routes too (indicated by the presence of two “routes” to each prefix and the asterisk at the start of each of the route lines).

BGP Testing

Now that we know that we have IP connectivity to the spines/route reflectors, we shouldn’t have any issues with BGP right?
We can confirm this from vtysh as well:

$ sudo vtysh

Hello, this is FRRouting (version 6.0.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

vagrant# show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 192.168.11.132, local AS number 65000 vrf-id 0
BGP table version 0
RIB entries 11, using 1760 bytes of memory
Peers 2, using 41 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
192.168.11.101  4      65000    1562    1562        0    0    0 00:10:55            3
192.168.11.102  4      65000    1562    1562        0    0    0 00:10:55            3

Total number of neighbors 2

Both neighbours are showing as up, which is great.

We’re also receiving prefixes from both peers (as indicated by the “3” under the header “State/PfxRcd”).
We can find what these are using show ip route bgp:

vagrant# show ip route bgp
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

B>  172.16.131.0/24 [200/0] via 192.168.11.131 (recursive), 00:12:01
  *                           via 192.168.11.113, enp0s8 onlink, 00:12:01
  *                           via 192.168.11.114, enp0s9 onlink, 00:12:01
B>  172.17.131.0/24 [200/0] via 192.168.11.131 (recursive), 00:12:01
  *                           via 192.168.11.113, enp0s8 onlink, 00:12:01
  *                           via 192.168.11.114, enp0s9 onlink, 00:12:01
B>  172.18.131.0/24 [200/0] via 192.168.11.131 (recursive), 00:12:01
  *                           via 192.168.11.113, enp0s8 onlink, 00:12:01
  *                           via 192.168.11.114, enp0s9 onlink, 00:12:01

This is mimicked by the kernel routing table, which we can look at by using iproute2’s “ip route show” command:

$ ip route show
default via 192.168.11.1 dev enp0s3 proto dhcp src 192.168.11.232 metric 100
172.16.131.0/24 proto bgp metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink
172.16.132.0/24 dev mgmt proto kernel scope link src 172.16.132.1 linkdown
172.17.131.0/24 proto bgp metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink
172.17.132.0/24 dev storage proto kernel scope link src 172.17.132.1 linkdown
172.18.131.0/24 proto bgp metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink
172.18.132.0/24 dev vxlan proto kernel scope link src 172.18.132.1 linkdown
192.168.11.0/24 dev enp0s3 proto kernel scope link src 192.168.11.232
192.168.11.1 dev enp0s3 proto dhcp scope link src 192.168.11.232 metric 100
192.168.11.101 proto ospf metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink
192.168.11.102 proto ospf metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink
192.168.11.111 proto ospf metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink
192.168.11.112 proto ospf metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink
192.168.11.113 via 192.168.11.113 dev enp0s8 proto ospf metric 20 onlink
192.168.11.114 via 192.168.11.114 dev enp0s9 proto ospf metric 20 onlink
192.168.11.131 proto ospf metric 20
	nexthop via 192.168.11.113 dev enp0s8 weight 1 onlink
	nexthop via 192.168.11.114 dev enp0s9 weight 1 onlink

Conclusion

In this post, we covered how to set up FRR on our OpenStack hosts in preparation of them advertising their VXLAN networks to each other.

In the next post, we’ll be preparing our hosts for deployment of OpenStack-Ansible.

References:

Ansible - “include_tasks” module
Ansible - “apt” module
Ansible - “reboot” module
Ansible - “include_role” module
Ansible - “template” module
Ansible - “service” module
ansible-frr role
FRR - Features and Kernel Support

Versions used:

Desktop Machine: kubuntu-18.04
VirtualBox: virtualbox-5.2.18
Vagrant: 2.2.3
Cumulus VX Vagrant Box: CumulusCommunity/cumulus-vx (virtualbox, 3.7.2)
Ubuntu Server Vagrant Box: geerlingguy/ubuntu1804 (virtualbox, 1.0.6)
Ansible: 2.7.6

OpenStack Lab Network - Host Network Interfaces

Sat, 26 Jan 2019 00:00:00 +0000

Overview

This post is the eighth in a series that plans to document my progress through installing and configuring a small OpenStack Lab.

For other posts in this series, see the overview section of the introduction post.

In the last post, we covered the routing configuration on our Cumulus switches.
In this post, we’ll go over the network interface configuration that our OpenStack hosts require in order to participate in the network that we have set up for them.

Configuration

In terms of networking on our OpenStack hosts, we want to accomplish a few things before we even think about installing and running OpenStack itself.
The first, and what we’ll cover in this post, will be configuring our network interfaces to support our routing protocols and our OpenStack configuration.

At this point, it might be important to remind ourselves that we’re preparing our hosts for an OpenStack deployment via OpenStack-Ansible. Their deployment guide contains tons of useful information when going through a first install.
Also of note for this series of posts is their routed environment example, as the configuration involved and the end state of the hosts is similar to what we want to deploy.

Ansible Playbook

As usual, our starting place is the playbook.
Because we’re no longer working with our Cumulus switches and instead on our OpenStack hosts themselves, let’s separate out any roles that we use into a new playbook. It’ll be called “openstack-hosts.yml”

---
- name: Configuring OpenStack hosts
  hosts: openstack_hosts
  become: true
  gather_facts: true
  roles:
    - role: openstack-interfaces
      tags:
        - openstack-interfaces

Ansible Role

I’ve already spoiled the name of the role we’ll create in the playbook above.
I’ll create the directory structure for it as usual.

$ mkdir -p /etc/ansible/roles/openstack-interfaces/tasks/

Let’s start adding tasks.

Packages

Although the OpenStack-Ansible documentation includes a list of required packages, we’ll not be installing all of them in this role.
In fact, the only package we are going to include from the documentation is “bridge-utils”, which allows the configuration of network bridges.

We’ve gone over installing packages in previous posts, so I won’t go into too much detail now.
Here’s the first task in our file “/etc/ansible/roles/openstack-interfaces/tasks/main.yml”

- name: Install prerequisite packages
  apt:
    name: bridge-utils
    state: present
    cache_valid_time: 3600

Let’s move on to our configuration.

Interfaces

Unlike our Cumulus switches, where our interfaces are configured via the file “/etc/network/interfaces” and brought up with the “networking” service, Ubuntu 18.04 implements a different tool for configuration called Netplan and brings this configuration to fruition through the newer “systemd-networkd” service.

Netplan is just another layer of configuration abstraction that sits on top of existing linux networking tools.
What this means for us is that we’re moving where we define our network interface configuration (“/etc/netplan/”), changing the format of this configuration to the more readable YAML (yes, the same configuration language that Ansible uses) and then calling Netplan to have this applied, through the appropriate tool, onto the network.

Let’s take a look at some example configuration that is already being used on our hosts.
The below configuration exists in the file “/etc/netplan/01-netcfg.yaml”, which is the default configuration file for Ubuntu 18.04 hosts:

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: yes

Pretty simple for the entire starting network configuration of our host right?
Let’s go over some core parts before we start our own configuration:

network: Top-level node. Always present.
version: The version of YAML being used.
renderer: The network daemon that Netplan will configure. At the time of writing two options are available - NetworkManager (which is what our Cumulus switches use) and Systemd-networkd (which is what our Ubuntu hosts use by default).
ethernets: A block specifying the type of interfaces being configured, under which new interfaces can be defined. Other types include wifis, bridges, bonds, tunnels and vlans.

Now that I’ve covered how Netplan works a little, I think it’s best if we dive right into our Ansible task and handlers.

Netplan configuration files live in “/etc/netplan” (as seen above) and must end in “.yaml” to be picked up.

---
- name: Configure Interfaces
  template:
    src: "interfaces_custom.j2"
    dest: "/etc/netplan/interfaces_custom.yaml"
  notify:
    - Apply Netplan configuration

This requires the presence of a handler file (“handlers/main.yml”) to run the command “netplan apply”, which tells Netplan to put our configuration file into action.

---
- name: Apply Netplan configuration
  command: netplan apply

I’ll cover the content of the template file in two parts, starting with the configuration of the loopback and physical interfaces on our hosts.

Loopback and Physical Interfaces

As with our Cumulus switches, our loopback and physical ports are there to support the routing topology over which OpenStack will communicate.
Our requirements for configuration will also remain the same, with our loopback interface needing an IP address so that BGP peering relationships can be established; and our physical interfaces using the same address for OSPF unnumbered (which I covered in my previous post).

I’m going to mix it up this time and define our variables before diving into our template.
We’re going to want to define two items as variables for both hosts - our loopback IP addresses and physical interface numbers.

Because our hosts share the same physical port layout (through our Vagrant/VirtualBox configuration), they both end up with the same network interface names for their physical ports (enp0s8 and enp0s9). We can, therefore, use the same variable for both of our hosts.
I’ll create this variable in the new file “/etc/ansible/group_vars/openstack_hosts/vars.yml”:

---
openstack_hosts_network_interfaces:
  - enp0s8
  - enp0s9

# OSPF Routing
openstack_hosts_routing_ospf_unnumbered: True
openstack_hosts_routing_ospf_enabled: True

# BGP Routing
openstack_hosts_routing_bgp_enabled: True
openstack_hosts_routing_bgp_evpn_enabled: True

You’ll see that I’ve also created some other variables that we used in our Cumulus interface and routing configurations. These let us say whether we’re using a given protocol on our hosts.

We also need to define the IP address that each host will use on its’ loopback interface.
Here’s what “/etc/ansible/host_vars/openstack-compute/vars.yml” looks like as an example:

---
openstack_host_loopback_address: "192.168.11.132/32"

Now that we’ve set ourselves up with some variables, let’s begin writing our template (“templates/interfaces_custom.j2”):

network:
  version: 2
  renderer: networkd
  ethernets:
    #######################
    # Loopback Interfaces #
    #######################
    lo:
      match:
        name: lo
{% if openstack_host_loopback_address is defined %}
      addresses:
        - {{ openstack_host_loopback_address }}
{% endif %}
    #######################
    # Physical Interfaces #
    #######################
{% for port in openstack_hosts_network_interfaces | sort %}
    {{ port }}:
{% if openstack_hosts_routing_ospf_unnumbered == true %}
      addresses:
        - {{ openstack_host_loopback_address }}
{% endif %}
      mtu: {{ openstack_lab_mtu }}
{% endfor %}

This template starts with the same configuration lines as the default one that we looked at above.
After we specify the interface type with “ethernets:”, we can define the interfaces that we want to bring up underneath.

The first of which is our loopback interface.
Because we are simply adding an IP address to an existing interface (“lo”), Netplan needs us to tell which interface we are adding configuration to.
This is done with the “match” and “name” keys, essentially telling Netplan to configure the interface that already exists with the name of “lo”.
We then use the “addresses” key followed by a list (of one in our case) of IP addresses that we want configured.

Our physical interfaces are very similar. Define the interface name, give it an IP address and then configure the MTU by using the “mtu” key.

OpenStack Bridges

As I eluded to before, I’m also going to include in this template the configuration of the network bridges required by OpenStack-Ansible.
This is simply so that we don’t need to configure this file again in another role.

OpenStack-Ansible utilises containers to operate the many services that are included in an OpenStack installation. To connect to the outside world, the containers need to talk via bridges that need to be set up on the host before deployment.
These bridges are as follows:

br-mgmt: Provides management of and communication between the infrastructure and OpenStack services.
br-storage: Provides access between OpenStack services and storage devices.
br-vlan: Provides internal communication between tenant networks that are VLAN tagged or flat (no VLAN tag) and external communication to outside of the OpenStack network through “Provider networks” (NAT).
br-vxlan: Only required if the environment is configured create virtual networks using VXLAN. Provides internal communication between tenant networks that use VXLAN tunnelling.

The documentation helps out here, by showing which bridges need to be present on a given host.
For our lab deployment, both hosts are going to be acting as network (control), compute and storage nodes.
This translates to needing all 4 bridges (br-mgmt, br-storage, br-vxlan and br-vlan) on both hosts, with IP addresses on all except br-vlan.

Like in our above step, let’s define variables first.
Because the IP addresses that we configure on these bridges are going to be different on each host, let’s define the variables under the “host_vars” hierarchy.
Here’s what I’ve added to “/etc/ansible/host_vars/openstack-compute/vars.yml” as an example:

openstack_host_network_bridges:
  mgmt: "10.131.16.1/24"
  storage: "10.131.17.1/24"
  vxlan: "10.131.18.1/24"
  vlan:

Note the blank value for the “vlan” key, because that bridge will not have an IP address configured.

These variables come together to form the below addition to our interface configuration template:

###########################
# Local Bridge Interfaces #
###########################
bridges:
{% for bridge, address in openstack_host_network_bridges.items() | sort %}
  {{ bridge }}:
    interfaces: []
    parameters:
      stp: false
      forward-delay: 0
{% if address is not none %}
    addresses:
      - {{ address }}
{% endif %}
{% endfor %}

Most of this configuration was done by looking at OpenStack-Ansible’s examples and translating this into to the Netplan syntax for configuration of bridges.

Let’s walk through the configuration of each bridge:

bridges: - Tells Netplan that the following interfaces are bridges.
{{ bridge }} - For every key that exists in the variable “openstack_host_network_bridges” for the host, create an interface with the name of the key with the following configuration underneath.
interfaces: [] - Usually we would assign network interfaces that are participating in the bridge here, but instead we say that there are none. OpenStack-Ansible is going to add ports to this bridge when it creates the appropriate containers.
parameters: - Tells Netplan to set special bridge options, depending on the nested keys and values.
stp: false - The first option we use to turn spanning tree off . This is because the bridges aren’t uplinking to any other switches (rather, this bridge will be advertised as a prefix by iBGP for reachability between hosts).
forward-delay: 0 - The second option says that there is to be no forwarding delay on the bridge (Tells the bridge to forward traffic immediately instead of going through the usual listening and learning stages).
If the bridge has an IP address specified, we set this using the standard syntax.

As simple as that, we’ve configured all prerequisite network interfaces.

Vagrant

The last thing we need to do is tell Vagrant that it needs to run the playbook after it has set up both OpenStack hosts.
I’ll accomplish this by including the provision block after the last OpenStack host, just like we did with the Cumulus provisioning block.

config.vm.define "openstack_compute" do |device|
  ...
  # Ansible Playbook for all OpenStack hosts
  device.vm.provision :ansible do |ansible|
    ansible.inventory_path = "/etc/ansible/hosts"
    ansible.playbook = "/etc/ansible/playbooks/openstack-hosts.yml"
    ansible.limit = "openstack_hosts"
  end
end

Testing

After all of our configuration, when we run vagrant up on our local machine, we get the following output after it has created the two OpenStack hosts:

$ vagrant up
...
==> openstack_compute: Running provisioner: ansible...
    openstack_compute: Running ansible-playbook...

PLAY [Configuring OpenStack hosts] *********************************************

TASK [Gathering Facts] *********************************************************
ok: [openstack-control]
ok: [openstack-compute]

TASK [openstack-interfaces : Install prerequisite packages] ********************
changed: [openstack-compute]
changed: [openstack-control]

TASK [openstack-interfaces : Configure Interfaces] *****************************
changed: [openstack-control]
changed: [openstack-compute]

RUNNING HANDLER [openstack-interfaces : Apply Netplan configuration] ***********
changed: [openstack-compute]
changed: [openstack-control]

PLAY RECAP *********************************************************************
openstack-compute          : ok=1    changed=3    unreachable=0    failed=0   
openstack-control          : ok=1    changed=3    unreachable=0    failed=0

Looks like everything was configured, let’s check that it’s taken effect on the hosts.

Interfaces

Using vagrant ssh $host we can connect to both of our hosts and find out whether everything looks as it should.

Firstly, let’s check whether our bridges exist:

$ brctl show
bridge name   bridge id           STP enabled   interfaces
mgmt          8000.ce995af87981   no		
storage       8000.864182527489   no		
vlan          8000.daeb42cdd17b   no		
vxlan         8000.000000000000   no

Although the command doesn’t show a lot of information (yet - brctl becomes a very useful command once MAC addresses are being learnt by each bridge), it does confirm that we’ve created the bridges on the host.

We can confirm this with the command ip address show:

$ ip address show | grep "UP\|inet" | grep -v "inet6"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    inet 127.0.0.1/8 scope host lo
    inet 192.168.11.132/32 brd 192.168.11.132 scope global lo:1
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.11.232/24 brd 192.168.11.255 scope global enp0s3
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.11.132/32 brd 192.168.11.132 scope global enp0s8
4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.11.132/32 brd 192.168.11.132 scope global enp0s9
5: vxlan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    inet 172.16.2.132/24 brd 172.16.2.255 scope global vxlan
6: vlan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
7: storage: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    inet 172.16.1.132/24 brd 172.16.1.255 scope global storage
8: mgmt: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    inet 172.16.0.132/24 brd 172.16.0.255 scope global mgmt

This command shows us each interface present and its’ associated IP address.

Conclusion

In this post, we covered some basic network interface configuration on our OpenStack hosts, including the bridges that OpenStack-Ansible needs present.

In the next post, we’ll build on top of this configuration by bringing up OSPF and iBGP.

Appendices

References

Ansible - “template” module
Ansible - “service” module
Ansible - “command” module
Ansible - “apt” module
Netplan - Reference Documentation
OpenStack Ansible - Deployment Guide
OpenStack Ansible - Routed Environment Example

Versions used

OpenStack Lab Network - Switch Configuration (Part 3)

Sun, 09 Dec 2018 00:00:00 +0000

Overview

This post is the seventh in a series that plans to document my progress through installing and configuring a small OpenStack Lab.

For other posts in this series, see the overview section of the introduction post.

In the last post, we covered some basic configuration of interfaces on a Cumulus switch.
This post will cover the last of our configuration required on our Cumulus switches - routing.

Configuration

First, I think it might be a good idea to recap what we want to be accomplished in terms of routing. Let’s take a look back to our production network diagram.

Our end goal with this topology is to have our OpenStack virtual machines able to talk to each other, even if they’re on the same subnet.
This can be accomplished by telling OpenStack to tag traffic with VLANs when it communicates across the network, which can work on a small scale quite well, but comes with the downsides of using VLANs as well such as requiring Spanning Tree be run to prevent loops from forming. It’s also not what I set out to learn about.
The other option is to tell OpenStack to use VXLANs to communicate over the network (Unfortunately at this stage OpenStack doesn’t yet support GENEVE as an option).

I won’t dive into the technical details of VXLAN in this post. Just know that VXLAN VTEPs (Virtual Tunnel Endpoints - The OpenStack hosts on our environment) send traffic by encapsulating received frames and tunnelling this across a layer 3 network to remote VTEPs. VTEPs need a way to learn what VNIs (Virtual Network Identifiers - VXLANs segment identifier, similar to a VLAN) are present in the network and on what hosts they reside on. They also need a mechanism to learn where MAC addresses within each VNI exist (What VTEP they terminate to).

VXLAN does have a native learning capability built in that uses multicast, but this has its shortcomings (like needing to run multicast in your network). That’s where EVPN comes in.
EVPN can act as a control plane for VXLAN and does so by advertising VXLAN information over BGP.

Okay, we have some requirements:

Have IP reachability across our network so that VXLAN can tunnel traffic.
Support the OpenStack nodes in advertising VXLAN information over BGP using EVPN.

I’ve decided to deploy OSPF as an underlay routing protocol. This will advertise loopback addresses between all hosts (switches and OpenStack nodes) for IP reachability.

Loopback reachability is also required for BGP, which we’ll be deploying on the spine switches in addition to the OpenStack nodes.
While the spine switches won’t be participating in the VXLAN network (they won’t have any VNI’s or MAC addresses to advertise), they will be acting as route reflectors.
The deployment of route reflectors isn’t needed in an environment this small but quickly becomes important in a larger network.

For further reading on other EVPN+BGP network designs, I would recommend taking a look at this post over on ipspace.net

That’s it for the design. Let’s jump into the configuration.

Ansible Playbook/Role

We’ve already created the playbook in a previous post, which references the “cumulus-base” and “cumulus-interface” roles. I’ll create a new one named “cumulus-routing”.

$ mkdir -p /etc/ansible/roles/cumulus-routing/tasks/

And add this to our playbook - “openstack-cumulus.yml”:

---
- name: Configuring Cumulus switches
  hosts: openstack_cumulus
  become: true
  gather_facts: true
  roles:
    - role: cumulus-base
      tags:
        - cumulus-base
    - role: cumulus-interface
      tags:
        - cumulus-interface
    - role: cumulus-routing
      tags:
        - cumulus-routing

Onto our role.

Free Range Routing

Free Range Routing (sometimes referred to as FRRouting or simply FRR) is an open source routing suite that Cumulus deploys on their switches. FRR manages the operations of any routing processes running on the switch, including OSPF and BGP.

FRR has a few main configuration files, stored under the directory “/etc/frr”:

$ ls -alh /etc/frr/
total 16K
drwxrwxr-x 1 frr  frr      74 Dec  8 13:52 .
drwxr-xr-x 1 root root   3.5K Dec  8 13:46 ..
-rw-r--r-- 1 frr  frr    1016 Nov  8 11:28 daemons
-rw-r--r-- 1 frr  frr    1.2K Nov  8 11:28 daemons.conf
-rw-r--r-- 1 frr  frr     120 Nov  8 11:28 frr.conf
-rw-r--r-- 1 frr  frrvty   60 Nov 14 07:30 vtysh.conf

Let’s quickly go over what each is used for:

File	Purpose
daemons	Controls what daemons (routing processes) are run. All daemons are disabled by default.
daemons.conf	Controls the options passed to daemons when they start (If they’re enabled in the above file).
frr.conf	Controls the configuration of all daemons. The meat of the routing configuration.
vtysh.conf	Controls how the command line interface utility (vtysh) operates.

In the context of the configuration we’re writing in this role, the files “daemons.conf” and “vtysh.conf” can be left alone.

That leaves us with needing to change the “daemons” and “frr.conf” files, which we’re going to do with the Ansible template module. Our “tasks/”main.yml” file ends up looking like the below for this role:

---
- name: Configure FRR (Free Range Routing)
  template:
    src: "{{ item }}.j2"
    dest: "{{ cumulus_frr_config_dir }}{{ item }}"
  loop:
    - daemons
    - frr.conf
  notify: Reload FRR

- name: Start and Enable FRR
  service:
    name: "{{ cumulus_frr_service_name }}"
    state: started
    enabled: yes

Note how we’ve only got two tasks:

Configure Free Range Routing.
Ensure that the FRR service is started (Because this is disabled by default).

We’ve also included a handler as a part of our configuration task above, so we need to define a “handlers/main.yml” file as well.

---
- name: Reload FRR
  service:
    name: "{{ cumulus_frr_service_name }}"
    state: reloaded

Lastly, we’ve referenced two new variables in the above files.
Because these variables are I’m going to use a new file, “defaults/main.yml”, and define the variables here.

---
# FRR Service Configuration
cumulus_frr_service_name: "frr"
cumulus_frr_config_dir: "/etc/frr/"

The defaults file allows for the definition of role-specific variables which take the standard value for that variable.
This allows for overwriting the variable in other locations (e.g. in group_vars/host_vars) if need be.
Note that a variable defined in this file is assumed to be of the lowest precedence, so any other definition of the variable overwrites it.

Now that we’ve done the groundwork, let’s get into our two jinja templates.

FRR Daemons

The first (and more simple) of the two templates that we’ll create is “/etc/ansible/roles/cumulus-routing/templates/daemons.j2”.
I’ll show you what this looks like and then explain what I’ve changed:

# {{ ansible_managed }}
# This file tells the frr package which daemons to start.
#
# Entries are in the format: <daemon>=(yes|no|priority)
#   0, "no"  = disabled
#   1, "yes" = highest priority
#   2 .. 10  = lower priorities
# Read /usr/share/doc/frr/README.Debian for details.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activation a daemon at the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
#
# The watchfrr daemon is always started. Per default in monitoring-only but
# that can be changed via /etc/frr/daemons.conf.
#
zebra={{ "yes"  if (cumulus_routing_bgp_enabled
                or cumulus_routing_ospf_enabled
                or cumulus_routing_ospf6_enabled
                or cumulus_routing_rip_enabled
                or cumulus_routing_ripng_enabled
                or cumulus_routing_isis_enabled
                or cumulus_routing_pim_enabled
                or cumulus_routing_ldp_enabled
                or cumulus_routing_nhrp_enabled
                or cumulus_routing_eigrp_enabled
                or cumulus_routing_babel_enabled
                or cumulus_routing_sharp_enabled
                or cumulus_routing_pbr_enabled)  else "no" }}
bgpd={{   "yes" if cumulus_routing_bgp_enabled   else "no" }}
ospfd={{  "yes" if cumulus_routing_ospf_enabled  else "no" }}
ospf6d={{ "yes" if cumulus_routing_ospf6_enabled else "no" }}
ripd={{   "yes" if cumulus_routing_rip_enabled   else "no" }}
ripngd={{ "yes" if cumulus_routing_ripng_enabled else "no" }}
isisd={{  "yes" if cumulus_routing_isis_enabled  else "no" }}
pimd={{   "yes" if cumulus_routing_pim_enabled   else "no" }}
ldpd={{   "yes" if cumulus_routing_ldp_enabled   else "no" }}
nhrpd={{  "yes" if cumulus_routing_nhrp_enabled  else "no" }}
eigrpd={{ "yes" if cumulus_routing_eigrp_enabled else "no" }}
babeld={{ "yes" if cumulus_routing_babel_enabled else "no" }}
sharpd={{ "yes" if cumulus_routing_sharp_enabled else "no" }}
pbrd={{   "yes" if cumulus_routing_pbr_enabled   else "no" }}

There are two halves to the above file, the lines of comments at the top of the file and then the setting of each daemon to “yes” or “no” in the bottom half.

We accomplish the setting of each daemon using Jinja inline if statements. If the variable we’re referencing is True then we set write “yes” (telling FRR to turn the daemon on), and if it isn’t then we write “no”.

The zebra daemon at the top of the list manages all of the other routing processes, including kernel routing table updates, interface lookups, and redistribution.
This needs to be enabled if any one of the individual protocol daemons is, and we accomplish this through multiple lines of or statements.

We’ll then define all of the above variables in “defaults/main.yml”, setting them to False by default.

---
# FRR Service Configuration
cumulus_frr_service_name: "frr"
cumulus_frr_config_dir: "/etc/frr/"

# FRR Daemons
cumulus_routing_bgp_enabled: False
cumulus_routing_ospf_enabled: False
cumulus_routing_ospf6_enabled: False
cumulus_routing_rip_enabled: False
cumulus_routing_ripng_enabled: False
cumulus_routing_isis_enabled: False
cumulus_routing_pim_enabled: False
cumulus_routing_ldp_enabled: False
cumulus_routing_nhrp_enabled: False
cumulus_routing_eigrp_enabled: False
cumulus_routing_babel_enabled: False
cumulus_routing_sharp_enabled: False
cumulus_routing_pbr_enabled: False

Lastly, because we want our switches to run OSPF and BGP, we’ll set “cumulus_routing_bgp_enabled” to True in the file “/etc/ansible/group_vars/openstack_cumulus/vars.yml”:

---
# Management
cumulus_management_interface: "eth0"

# OSPF Routing
cumulus_routing_ospf_unnumbered: True
cumulus_routing_ospf_enabled: True

# BGP Routing
cumulus_routing_bgp_enabled: True

FRR Routing Configuration

The second template that we’re creating is “/etc/ansible/roles/cumulus-routing/templates/frr.conf.j2”.
This file will end up looking like the output of the traditional show running-configuration command on a Cisco router, as the syntax is very similar.

I’m going to split the writing of this file into many separate parts, as it’ll be easier to explain that way.

Standard FRR Configuration

We’ll start with the top of this template:

! {{ ansible_managed }}
!
frr defaults datacenter
!
hostname {{ inventory_hostname }}
username cumulus nopassword
!
service integrated-vtysh-config
!
log syslog informational
!
line vty
!
router-id {{ cumulus_host_loopback_address | ipaddr('address') }}
!

We’re just setting some base daemon information here:

frr defaults datacenter - Configuring routing process defaults
hostname - What the hostname of the router is, using the “inventory_hostname” variable.
username - The base user.
service integrated-vtysh-config - Tells FRR to use one config file, frr.conf.
log syslog informational - Output all logs to “/var/log/frr/frr.log”.
line vty - Configuration of access to the command line interface. Note that this is not configuration of a physical console port, but the shell access through vtysh.
router-id - Sets the global router ID for the switch. As this should not include the subnet CIDR notation, we strip this out using the Jinja ipaddr filter.

Nothing too fancy. Let’s move on.

OSPF Unnumbered

OSPF Unnumbered is a way for network devices to talk OSPF to each other over interfaces that don’t themselves have unique IP addresses.
I say unique because the interfaces still have IP addresses configured on them, but in an unnumbered environment all interfaces can take the same IP address (the loopback for example).

We did exactly that in the previous post, where we configured the loopback address of each host and configured this on each switchport interface as well.
This tells the Cumulus switches to use the loopback address for OSPF messages on the link instead of unique addressing, making for easier configuration and reduces the OSPF database size as there aren’t individual link addresses for each switchport.

The one caveat is that unnumbered interfaces are only supported on OSPF point-to-point links.
Let’s make sure we don’t miss that by adding the below configuration into our template:

!
{# Configure switchports as OSPF point-to-point #}
{% if cumulus_routing_ospf_enabled == true %}
{% for port in cumulus_switchgroup_switchports | sort %}
interface {{ port }}
 ip ospf network point-to-point
!
{% endfor %}
{% endif %}
!

The logic on this one is pretty simple. As long as OSPF is enabled, set each switchport to OSPF point-to-point.

OSPF Underlay

Again, we’ll jump into the configuration and then cover what it does.

!
{% if cumulus_routing_ospf_enabled == true %}
{# Configure switchports as OSPF point-to-point #}
{% for port, value in cumulus_switchgroup_switchports.items() | sort %}
interface {{ port }}
  ip ospf network point-to-point
!
{% endfor %}
!
router ospf
  ospf router-id {{ cumulus_host_loopback_address | ipaddr('address') }}
  network {{ cumulus_host_loopback_address }} area 0.0.0.0
  passive-interface lo
{% endif %}
!

Because we’re including the underlay configuration inside of the if loop we created before, we don’t need to check again if OSPF should be enabled. So we jump straight into router ospf configuration hierachy.

ospf router-id defines the router ID that the OSPF process will use.
Side note - Because we’ve defined this globally we don’t need to do this, but it’s good practice to hardcode this in the case the global router-id changes for any reason.

The network configuration line accomplishes two things. Firstly, it indicates to OSPF that it can start sending and listening for messages on all interfaces that have the loopback address configured (the loopback interface and, with our configuration in the previous section, all switchports too). This also tells OSPF to advertise the loopback address to its neighbours in area “0.0.0.0”. This is required for IP reachability between each device and ultimately enables BGP peerings to be established from the OpenStack hosts to the spine switches.

Finally, passive-interface lo disables OSPF from sending and listening for messages on the loopback interface.

And that’s it for OSPF. Easy huh?
Other changes that might be applicable to a larger production environment would include:

Configuring a different OSPF area for each rack and summarizing on the leaf switches.
Configuring border leaf switches to advertise external/default routes into OSPF.
Configuring authentication for OSPF interfaces so that unwanted speakers can’t participate.

iBGP Overlay

Here’s the configuration:

{% if cumulus_routing_bgp_enabled == true and ansible_hostname in groups['openstack_cumulus_spines'] %}
router bgp {{ ibgp_autonomous_system }} view iBGP-RR
  bgp router-id {{ cumulus_host_loopback_address | ipaddr('address') }}
  bgp cluster-id 1
{# Defines a peer group where common configuration to be defined for all peers #}
  neighbor iBGP-RR-Clients peer-group
  neighbor iBGP-RR-Clients remote-as internal
{# Forms a neighborship with the other spine switches #}
{% for spine in groups['openstack_cumulus_spines'] | difference([inventory_hostname]) %}
  neighbor {{ hostvars[spine].cumulus_host_loopback_address }} peer-group iBGP-RR-Clients
{% endfor %}
{# Enables the BGP speaker to peer with all hosts within a network range #}
  bgp listen range {{ cumulus_spines_bgp_RR_network }} peer-group iBGP-RR-Clients
{% if cumulus_routing_bgp_evpn_enabled == true %}
{# Enables advetisement of EVPN address family #}
  address-family l2vpn evpn
    neighbor iBGP-RR-Clients activate
    neighbor iBGP-RR-Clients route-reflector-client
  exit-address-family
{% endif %}
{% endif %}

This one might take a little bit to explain. We’ll take it one step at a time.

Just like the other sections, we want to check whether BGP should be configured on the host.
This time it isn’t enough to check whether “cumulus_routing_bgp_enabled” has been set because we only need to configure BGP on the spine switches.
The second check is completed by asking whether the current host the Ansible role is running against is in the host group “openstack_cumulus_spines”.

If the host clears the if logic, we start by entering into the router bgp hierachy by stating the AS number and view.
Because we’re running iBGP between the spine switches and the OpenStack hosts, the AS number will be the same between them. We’ll define this variable shortly.
BGP views are something that I hadn’t come across until writing this post. Simply put, BGP processes configured with a view participate in BGP like any other speaker, with the only difference being that routes learnt in the view process are not installed into the kernel routing table (and therefore the forwarding table) of the device.
This is perfect for spine switches, as they won’t need to install the EVPN routes learnt from the OpenStack clients, freeing up resources for other uses.

bgp router-id simply tells BGP which ID to use when communicating. We’ll set that to the loopback address, just like we did with OSPF.

Setting bgp cluster-id configures the BGP process to act as a route reflector for any devices that peer with it.
I’ve decided to use the same cluster ID on both of our route reflectors (spine switches), but it is possible to configure different a ID on each. Orhan Ergun writes a nice post on the topic of when to use unique cluster IDs.
In a topology like ours where our servers are the route reflector clients instead of the leaf switches, the risk of losing reachability between clients due to not knowing about the other RR’s routes is decreased due to each host connecting to two upstream leaf switches - therefore having 4 paths to each RR (and each other OpenStack node in another “rack”) instead of just the two.
In an OpenStack EVPN environment, where the route reflectors are going to be learning routes for every MAC address present, my recommendation would be to configure the cluster-id the same on each RR. You can then size up how many entries are present in the route table and determine whether having more than one copy is possible.

The next two lines create a BGP neighbour group named “iBGP-RR-Clients”. Neighbour groups allow for easier configuration of neighbours with common settings.
The peer-group line simply defines the neighbour group.
The remote-as internal line tells BGP that all neighbours in the group will be iBGP neighbours.

The next for loop, setting the line neighbor ... peer-group iBGP-RR-Clients, configures a BGP neighborship manually with the other spine switches in the topology (in our case, just the other 1).

bgp listen range allows the BGP speakers to form peerings with any host in the defined network.
For this, we’re referencing a new variable which will house the range that we expect the clients (OpenStack hosts) loopbacks to be inside - 192.168.11.0/24.

The last lines tell the route reflectors to participate in the IPv4 and EVPN address family so that IP address and VXLAN information can be transferred between clients.

We’ve referenced a few new variables in the above configuration. Let’s define these now.

“cumulus_routing_bgp_evpn_enabled” will have a default of “False”, but needs to be enabled for our environment.
In our role defaults:

...
# FRR Routing Configuration
cumulus_routing_bgp_evpn_enabled: False

And “/etc/ansible/group_vars/openstack_cumulus/vars.yml”:

...
# BGP Routing
cumulus_routing_bgp_enabled: True
cumulus_routing_bgp_evpn_enabled: True

I’ve placed the iBGP autonomous system number variable in “/etc/ansible/group_vars/openstack_lab/vars.yml” so that all hosts (switches and OpenStack servers) can access it:

...
# IP Routing
ibgp_autonomous_system: 65000

Lastly, our route reflector client network only needs to be accessible to the spine switches.
In “/etc/ansible/group_vars/openstack_cumulus_spines/vars.yml”:

...
# BGP routing
cumulus_spines_bgp_RR_network: "192.168.11.0/24"

Testing

Now that we’re done with our configuration we can deploy this to our hosts and test that it is working.

$ ansible-playbook /etc/ansible/playbooks/openstack-cumlus.yml
...
PLAY RECAP ****************************************************************
openstack-cumulus-leaf01   : ok=11   changed=5    unreachable=0    failed=0
openstack-cumulus-leaf02   : ok=11   changed=5    unreachable=0    failed=0
openstack-cumulus-leaf03   : ok=11   changed=5    unreachable=0    failed=0
openstack-cumulus-leaf04   : ok=11   changed=5    unreachable=0    failed=0
openstack-cumulus-spine01  : ok=11   changed=5    unreachable=0    failed=0
openstack-cumulus-spine02  : ok=11   changed=5    unreachable=0    failed=0

We can then log into our switches (using the tool vagrant ssh) and check that everything still looks okay.
As an initial check, let’s look at how the FRR service is doing:

systemctl status frr
● frr.service - FRRouting
   Loaded: loaded (/lib/systemd/system/frr.service; enabled)
   Active: active (running) (Result: exit-code) since Sun 2018-12-09 15:18:32 NZDT; 30s ago
  Process: 10291 ExecReload=/usr/lib/frr/frr-reload.py --reload /etc/frr/frr.conf (code=exited, status=0/SUCCESS)
  Process: 6500 ExecStart=/usr/lib/frr/frr start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/frr.service
           ├─6517 /usr/lib/frr/zebra -M snmp -s 90000000 --daemon -A 127.0.0.1
           ├─6524 /usr/lib/frr/bgpd -M snmp --daemon -A 127.0.0.1
           ├─6533 /usr/lib/frr/ospfd -M snmp --daemon -A 127.0.0.1
           └─6539 /usr/lib/frr/watchfrr -d -r /usr/sbin/servicebBfrrbBrestartbB%s -s /usr/sbin/servicebBfrrbBstartbB%s -k /usr/sbin/servicebBfrrbBstopbB%s -b bB zebra bgpd ospfd

There are a few important details in the output above.
First (and most importantly) the service started successfully and is now running.
Also, under the control group (CGroup) heading we can see that several processes have spawned. “zebra”, “bgpd” and “ospfd” are present as expected and the watchdog watchfrr (which we haven’t talked about yet) is running as well.

OSPF Testing

In terms of checking the status of network connectivity and routing protocols, Cumulus switches give us two options:

Utilize Cumulus’ “NCLU” command line utility.
Login into FRR’s console shell (vtysh).

Because NCLU simply piggybacks on top of vtysh, I’ll only be showing the NCLU commands in this section.
I’ll be performing these checks from the perspective of leaf #1.

The first OSPF check should be whether our interface configuration has been picked up.
To complete this with NCLU, we type in commands from the main bash shell of the devices:

$ net show ospf interface | grep "lo is\|swp\|Area\|Network\|Neighbor"
lo is up
  Internet Address 192.168.11.111/32, Area 0.0.0.0
  Router ID 192.168.11.111, Network Type LOOPBACK, Cost: 10
  Neighbor Count is 0, Adjacent neighbor count is 0
swp1 is up
  This interface is UNNUMBERED, Area 0.0.0.0
  Router ID 192.168.11.111, Network Type POINTOPOINT, Cost: 100
  Neighbor Count is 1, Adjacent neighbor count is 1
swp2 is up
  This interface is UNNUMBERED, Area 0.0.0.0
  Router ID 192.168.11.111, Network Type POINTOPOINT, Cost: 100
  Neighbor Count is 1, Adjacent neighbor count is 1
swp3 is up
  This interface is UNNUMBERED, Area 0.0.0.0
  Router ID 192.168.11.111, Network Type POINTOPOINT, Cost: 100
  Neighbor Count is 0, Adjacent neighbor count is 0

From here we can see that all of our interfaces are up, that they’re using the loopback interfaces address and how many neighbours are present on each.
Note how there aren’t any neighbours present on switchport 3, as this is a downlink to an OpenStack machine that isn’t configured yet.

Next up is confirming the neighbourships.

$ net show ospf neighbor

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
192.168.11.101    1 Full/DROther      30.670s 192.168.11.101  swp1:192.168.11.111      0     0     0
192.168.11.102    1 Full/DROther      30.484s 192.168.11.102  swp2:192.168.11.111      0     0     0

As expected, two neighbourships - one to each spine switch.

Let’s also take a look at the OSPF routing table, as that should now be populated with the loopback addresses of each switch in the network:

$ net show ospf route
============ OSPF network routing table ============
N    192.168.11.101/32     [100] area: 0.0.0.0
                           via 192.168.11.101, swp1
N    192.168.11.102/32     [100] area: 0.0.0.0
                           via 192.168.11.102, swp2
N    192.168.11.111/32     [0] area: 0.0.0.0
                           directly attached to lo
N    192.168.11.112/32     [200] area: 0.0.0.0
                           via 192.168.11.101, swp1
                           via 192.168.11.102, swp2
N    192.168.11.113/32     [200] area: 0.0.0.0
                           via 192.168.11.101, swp1
                           via 192.168.11.102, swp2
N    192.168.11.114/32     [200] area: 0.0.0.0
                           via 192.168.11.101, swp1
                           via 192.168.11.102, swp2

It’s important to check that we’re receiving two routes for each other leaf switch’s loopback address as that confirms that each spine is receiving and readvertising them.

Lastly, we’ll check the routing table to confirm that the routes above have been installed.

$ net show route ipv4
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR,
       > - selected route, * - FIB route

K>* 0.0.0.0/0 [0/0] via 192.168.11.1, eth0, 02:21:44
C>* 192.168.11.0/24 is directly connected, eth0, 02:21:44
O>* 192.168.11.101/32 [110/100] via 192.168.11.101, swp1 onlink, 00:39:49
O>* 192.168.11.102/32 [110/100] via 192.168.11.102, swp2 onlink, 02:21:24
C * 192.168.11.111/32 is directly connected, swp3, 02:21:45
C * 192.168.11.111/32 is directly connected, swp2, 02:21:45
C * 192.168.11.111/32 is directly connected, swp1, 02:21:45
O   192.168.11.111/32 [110/0] is directly connected, lo, 02:21:45
C>* 192.168.11.111/32 is directly connected, lo, 02:21:45
O>* 192.168.11.112/32 [110/200] via 192.168.11.101, swp1 onlink, 00:39:49
  *                             via 192.168.11.102, swp2 onlink, 00:39:49
O>* 192.168.11.113/32 [110/200] via 192.168.11.101, swp1 onlink, 00:39:39
  *                             via 192.168.11.102, swp2 onlink, 00:39:39
O>* 192.168.11.114/32 [110/200] via 192.168.11.101, swp1 onlink, 00:39:39
  *                             via 192.168.11.102, swp2 onlink, 00:39:39

$ ip route
default via 192.168.11.1 dev eth0
192.168.11.0/24 dev eth0  proto kernel  scope link  src 192.168.11.211
192.168.11.101 via 192.168.11.101 dev swp1  proto ospf  metric 20 onlink
192.168.11.102 via 192.168.11.102 dev swp2  proto ospf  metric 20 onlink
192.168.11.112  proto ospf  metric 20
	nexthop via 192.168.11.101  dev swp1 weight 1 onlink
	nexthop via 192.168.11.102  dev swp2 weight 1 onlink
192.168.11.113  proto ospf  metric 20
	nexthop via 192.168.11.101  dev swp1 weight 1 onlink
	nexthop via 192.168.11.102  dev swp2 weight 1 onlink
192.168.11.114  proto ospf  metric 20
	nexthop via 192.168.11.101  dev swp1 weight 1 onlink
	nexthop via 192.168.11.102  dev swp2 weight 1 onlink

The top command shows FRR’s view of the kernel routing table, where we can see that routes are present.
Importantly, we can see that both routes received to leaf switch loopbacks are installed, to be used for ECMP load balancing. We can confirm that Linux is thinking the same way by running the ip route show command (or just ip route for short).

BGP Testing

Unfortunately, we can’t check much on the BGP front at this stage, as our spine switches don’t have any clients peered yet.
We’ll cover some good BGP (and EVPN) checks in a later post where we configure FRR on the OpenStack hosts.

Conclusion

In this (quite lengthy - sorry!) post, we’ve configured the routing protocols needed for our Cumulus switches to support the OpenStack network.

In the next post, I’ll start to dive into the network interface configuration required on our OpenStack hosts, before following up with the routing configuration using FRR.

Appendices

References

Ansible - “template” module
Ansible - “service” module
Jinja - Inline if Statements
Jinja - Multi-line expressions
Cumulus Linux - FRR Configuration
Cumulus Linux - OSPF
Cumulus Linux - ECMP Free Range Routing - User Guide
Orhan Ergun - BGP Route Reflector Clusters
RFC7348 - VXLAN
RFC8365 - Network Virtualization Overlay Using EVPN

Versions used

Desktop Machine: kubuntu-18.04
VirtualBox: virtualbox-5.2.18
Vagrant: 2.2.2
Cumulus VX Vagrant Box: CumulusCommunity/cumulus-vx (virtualbox, 3.7.2)
Ansible: 2.7.4

OpenStack Lab Network - Switch Configuration (Part 2)

Fri, 23 Nov 2018 00:00:00 +0000

Overview

This post is the sixth in a series that plans to document my progress through installing and configuring a small OpenStack Lab.

For other posts in this series, see the overview section of the introduction post.

In the last post, we finished the base configuration of our Cumulus switches using Ansible. This post will cover the configuration of network interfaces on Cumulus switches.

Configuration

For me, configuring the interfaces on a network device isn’t just enabling the required physical ports.
I think of this stage as laying the framework that our routing configuration will sit on top of.

What do we need to do before we bring up OSPF or BGP? I can think of a few things:

Configuration of the loopback interface
Setting the physical properties of required ports
Setting the MTU of all interfaces
Naming/Describing all interfaces
Configuring IP addresses on interfaces
Bringing up LLDP

Outside of our topology, you might also include some other steps. For example:

Creation of VLANs/Bridges
Configuration of LAGs (Including MCLAG)
Configuration of STP
Configuring and testing Cumulus PTM
Configuring 802.1X for port authentication

Ansible Playbook/Role

We’ve already created the playbook in the previous post, which references the “cumulus-base” role.
All we need to do now is create a new role for the interface configuration, “cumulus-interface”, and reference this in our existing playbook.

Creating the role directory:

$ mkdir -p /etc/ansible/roles/cumulus-interface/tasks/

And adding this to our playbook - “openstack-cumulus.yml”:

---
- name: Configuring Cumulus switches
  hosts: openstack_cumulus
  become: true
  gather_facts: true
  roles:
    - role: cumulus-base
      tags:
        - cumulus-base
    - role: cumulus-interface
      tags:
        - cumulus-interface

I’d like to make a quick digression here.
While we could have made only one role, named something like “cumulus-configure”, I believe that it is better to try and separate out the functions of roles.

I can think of two examples of why this might be useful

Say I wanted to include the base configuration of the switches in the ZTP script instead. With separate roles this is as easy as removing the role from the playbook, whereas with a single role you would need to manually prune the tasks in the role.
In the case that you wanted to deploy an utterly different topology onto the switches, where the difference in interface configuration would be too complex to bundle into a single role, you can simply rename the existing role to “cumulus-interface-openstack” and create a new role for the new topology.
(It’s important to think about the tradeoff you’re making with this reduction in complexity because you might end up making more work in the future maintaining two roles instead of just the one.)

Cumulus Interface Configuration

For network interface configuration, Cumulus again follows closely to how Debian does it.

On Debian, network interface configuration is held in the file “/etc/network/interfaces”. Optionally, this file can also include (“source” is the appropriate configuration term) configuration files, usually held in the directory “/etc/network/interfaces.d/”.
The Debian network interface management tool “ifupdown” is then run (with this file as an input) to configure the interfaces.

The only difference when it comes to Cumulus is that “ifupdown” is replaced with their own implementation - “ifupdown2”.

Let’s have a look at the default “/etc/network/interfaces” file on a Cumulus box:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*.intf

# The loopback network interface  
auto lo
iface lo inet loopback

# The primary network interface  
auto eth0
iface eth0 inet dhcp

I’ll also check whether there are any files already in “/etc/network/interfaces.d/”

$ ls -alh /etc/network/interfaces.d/
total 0
drwxr-xr-x 1 root root   0 Nov  7 09:56 .
drwxr-xr-x 1 root root 132 Nov 14 07:47 ..

Now that we know where to configure our interfaces, let’s jump into our first task.

Loopback Interface Configuration

As you can see in the above section, the loopback interface is already defined by default.
However, this interface doesn’t have an IP address - which is needed for connectivity in our implementation of BGP.
Let’s change that.

In the new role, under “tasks/main.yml”, I’ve created the following task:

---
- name: Configure Interfaces
  template:
    src: "interfaces_custom.j2"
    dest: "/etc/network/interfaces.d/interfaces_custom.intf"
  notify:
    - Restart networking

The jinja template file (“templates/interfaces_custom.j2”) looks like the following:

# {{ ansible_managed }}

######################
# Loopback Interface #
######################
auto lo
iface lo inet loopback
{% if cumulus_host_loopback_address is defined %}
    address {{ cumulus_host_loopback_address }}
{% endif %}

Because of the above, I’m also going to create the variable “cumulus_host_loopback_address” for all of our Cumulus hosts.
Because each switch will have a different loopback address, I’m going to create six different host variable files (one for each switch) and create the variable for each.
Here’s what “/etc/ansible/host_vars/cumulus-spine01/vars.yml” looks like as an example:

---
cumulus_host_loopback_address: "192.168.11.101/32"

Finally, we need to restart the “networking” service on the switches if there are any changes made.
Here’s what my “handlers/main.yml” file looks like:

- name: Restart networking
  service:
    name: networking
    state: restarted

Physical Interface Configuration

Physical network configuration is completed in the same way as the loopback, just with different naming and a few more options to consider.
The ones that we’re going to start with are “auto port” and “iface port”, which both need to be present for ifupdown to load a given port.

Because we’ve already written the task to copy our interfaces template file into the sourced directory, we simply need to append the physical interface configuration to our existing template:

#########################
# Switchport Interfaces #
#########################
{% for port, value in cumulus_switchgroup_switchports.items() | sort %}
auto {{ port }}
iface {{ port }}

{% endfor %}

If we remember back to when the Ansible hosts file was created, the groups “openstack_cumulus_spines”, “openstack_cumulus_leafgroup1” and “openstack_cumulus_leafgroup2”. This was because the switches in each group have something in common - every switch connects to the same devices.
As an example, the leaf switches 01 and 02 both have one uplink port to each spine switch and one downlink port to the connected OpenStack host.

Before we dive into the logic of the above template, I think it might be better to show you the variable files that I’m creating for each group first.
Here’s what the “/etc/ansible/group_vars/openstack_cumulus_spines/vars.yml” looks like:

---
cumulus_switchgroup_switchports:
  swp1:
    alias: downlink-cumulus-leaf01
  swp2:
    alias: downlink-cumulus-leaf02
  swp3:
    alias: downlink-cumulus-leaf03
  swp4:
    alias: downlink-cumulus-leaf04

And “/etc/ansible/group_vars/openstack_cumulus_leafgroup1/vars.yml”:

---
cumulus_switchgroup_switchports:
  swp1:
    alias: uplink-cumulus-spine01
  swp2:
    alias: uplink-cumulus-spine02
  swp3:
    alias: downlink-openstack-control01

Now back to the logic of the template.

For every key (ports/”swpX” entries) in the dictionary “cumulus_switchgroup_switchports”, and their associated values:
- First sort them, so that they are being iterated over in numerical order.
- Write the key string into the file, first after “auto” and again after “iface”.

This might look a little more complicated than it needs to be, and you would be right if you thought this way. However, this is because we’re going to be expanding on this loop in the next configuration steps.

MTU

MTU on Cumulus needs to be configured under each interface individually - unless you’re okay with using the default of 1500.
This is set by adding an indented line of “mtu value” under the “iface port” line for the port.

Now that we have a base template for each switch port, this shouldn’t be too hard.
First, let’s create a variable for the MTU, which will be shared across all ports.
In “/etc/ansible/group_vars/openstack_lab/vars.yml”, we’ll make the variable “openstack_lab_mtu”:

...
# Network Interfaces
openstack_lab_mtu: 9000

Next is to include it in the template:

#########################
# Switchport Interfaces #
#########################
{% for port, value in cumulus_switchgroup_switchports.items() | sort %}
auto {{ port }}
iface {{ port }}
    mtu {{ openstack_lab_mtu }}
{% endfor %}

Note that because this is a global variable we didn’t need to include this in each switch group’s variable file.
Likewise, the reference in the template is to the global variable and not to either the port or value in the loop.

Interface Descriptions

Interface descriptions in Debian network configuration are referred to as “aliases”, and configured as such.
Like with the MTU for a port, the alias is configured with another indented line under “iface port”, this time with “alias value”.

If you’ve been observant, you’ll notice that we have already created a key named “alias” under each switch port in the switch group variables.
That way we can use this in our template, which now becomes:

#########################
# Switchport Interfaces #
#########################
{% for port, value in cumulus_switchgroup_switchports.items() | sort %}
auto {{ port }}
iface {{ port }}
    mtu {{ openstack_lab_mtu }}
{% if 'alias' in value %}
{# Description of the interface #}
    alias {{ value.alias }}
{% endif %}

{% endfor %}

Let’s come back to the logic for the loop again and expand on it to cover the alias inclusion.

For every key (ports/”swpX” entries) in the dictionary “cumulus_switchgroup_switchports”, and their associated values:
- First sort them, so that they are being iterated over in numerical order.
- Write the key string into the file, first after “auto” and again after “iface”.
- Write “mtu” followed by what the variable “openstack_lab_mtu” is set to.
- If the key has a value inside of it with the name of “alias” then write “alias” followed what the value is set to.

Interface IP Addresses

The last interface configuration change we’ll make is to add an IP address to each interface.
This won’t be just any IP address though, as we’re going to reuse the loopback address so that we can take advantage of OSPF unnumbered.

I’ll cover this in detail in the next post on routing configuration.

#########################
# Switchport Interfaces #
#########################
{% for port, value in cumulus_switchgroup_switchports.items() | sort %}
auto {{ port }}
iface {{ port }}
    mtu {{ openstack_lab_mtu }}
{% if cumulus_routing_ospf_unnumbered == true %}
    address {{ cumulus_host_loopback_address }}
{% endif %}
{% if 'alias' in value %}
{# Description of the interface #}
    alias {{ value.alias }}
{% endif %}

{% endfor %}

In “/etc/ansible/group_vars/openstack_cumulus/vars.yml”, we’ll make the variable “cumulus_routing_ospf_unnumbered” and set this to True:

---
# Management
cumulus_management_interface: "eth0"

# Routing
cumulus_routing_ospf_unnumbered: True

Configuring LLDP

LLDP on Cumulus is implemented with lldpd and is enabled by default on all interfaces. Therefore, all I’m going to configure in this step is what interfaces LLDP should not run on.
This is done with the file “/etc/lldpd.d/exclude.conf”:

- name: Configure LLDP
  template:
    src: lldpexclude.conf.j2
    dest: /etc/lldpd.d/exclude.conf
  notify: Restart LLDP

This is a pretty simple file, with just one line of configuration:

# {{ ansible_managed }}
configure system interface pattern-blacklist {{ cumulus_management_interface }}

And the handler simply restarts the “lldp” service:

- name: Restart LLDP
  service:
    name: lldpd
    state: restarted

If we were deploying this into a production environment, I would probably want to add Cumulus PTM configuration to this step as well.
As this is a lab, where I might be changing connections at any time, I’m going to leave this for now.

Conclusion

In this post, I covered some basic configuration of interfaces on a Cumulus switch.

In the next post, I’ll start to dive into the routing configuration required for our OpenStack deployment.

Appendices

References

Ansible - “template” module
Ansible - “service” module
Debian Network Configuration
Cumulus Linux - Interface Configuration
Cumulus Linux - Switchport Attributes
Cumulus Linux - OSPF Unnumbered
Cumulus Linux - LLDP

Versions used

Desktop Machine: kubuntu-18.04
VirtualBox: virtualbox-5.2.10
Vagrant: 2.2.1
Cumulus VX Vagrant Box: CumulusCommunity/cumulus-vx (virtualbox, 3.7.2)
Ansible: 2.7.2

OpenStack Lab Network - Switch Configuration (Part 1)

Sun, 18 Nov 2018 00:00:00 +0000

Overview

This post is the fifth in a series that plans to document my progress through installing and configuring a small OpenStack Lab.

For other posts in this series, see the overview section of the introduction post.

In the last post, we configured the HTTP part of the ZTP server and then showed that we were able to bring everything up with Vagrant. This post will cover the base configuration of the Cumulus switches using Ansible.

Configuration

So what constitutes “base” configuration?
When I think of ‘base’, I think of everything that is done before any network configuration is made:

Installing any prerequisite/additional packages
~~Install the licence~~ (Not relevant for Cumulus VX)
Configuring DNS (Including hostname)
Configuring NTP
Configuring Authentication

The first one is definitely a new one for network devices as most network operating systems wouldn’t allow for any packages to be installed, but as the switch is a Linux device I’m going to install some packages that I like.

Ansible Playbook

We’ll start by writing the playbook that we’ll call.
In the directory “/etc/ansible/playbooks/” I’m creating another file, this time with the name “openstack-cumulus.yml”:

---
- name: Configuring Cumulus switches
  hosts: openstack_cumulus
  become: true
  gather_facts: true
  roles:
    - role: cumulus-base
      tags:
        - cumulus-because

Super simple. We want to run a single role (“cumulus-base”, which we’ll create next) on all of our switches to configure them.

Ansible Role

To start, we’ll create the roles “tasks” directory:

$ mkdir -p /etc/ansible/roles/cumulus-base/tasks/

Within this directory, let’s make a “main.yml” file and start adding tasks.

Installing packages

The first thing listed is the installation of packages.
Cumulus runs on top of Debian Jessie and uses “apt” as its’ package manager, so we’ll use the apt Ansible module for this step.

With the apt module, I like to start with a base state of declaring the name of each package and then the state that I want Ansible to keep it at.
I think this is a good habit to get into as it reminds me to think about whether I want a given package to remain stable or not.

The only other option I’ll include is cache_valid_time and set this to something a larger than the time between test runs of a playbook (I’ve found 15 minutes to be a good place to set this) so that the task isn’t running every time I test.

I’m just going to install some troubleshooting packages, “bwm-ng” and “mtr”, but because we added the Debian repositories to apt as a part of our ZTP script we could install any others that we saw fit.

---
- name: Install apt packages
  apt:
    name: "{{ item.name }}"
    state: "{{ item.state }}"
    cache_valid_time: 900
  loop:
    - name: bwm-ng
      state: latest
    - name: mtr
      state: latest
  tags:
    - packages

Configuring DNS

Because we can skip installing a licence (Cumulus VX doesn’t require one), next on our list is DNS configuration.

DNS configuration on a switch means

Setting DNS Servers
Setting the hostname of the device

We can (again) skip a step, in setting the DNS servers, this time because our switch receives these via DHCP when it boots.

To set the hostname with Ansible we use the hostname module and pass it the hostname that we’ve entered into our Ansible “hosts” file (another reason why this is important).

- name: Set hostname
  hostname:
    name: "{{ inventory_hostname }}"

Configuring NTP

NTP configuration consists of setting the timezone and then the NTP servers to get the time from.
On Debian, both of these are set within files and then the appropriate service reloaded.

Configuring Timezone

This one is nice and easy as the file we’re changing, “/etc/timezone/”, consists of a single line that is set to a Continent/City pair string.
For me, that’s “Pacific/Auckland”.

Because this is a text file, we can set this in multiple ways with Ansible.

copy module using the content option.
template module and a source jinja template.

As we’ll see going forward, the copy option would end up being incredibly messy for some of the longer files. This paired with the idea that it might be nice to use the same module throughout all of our configuration tasks means that I’ll go with template instead.

- name: Set Timezone
  template:
    src: timezone.j2
    dest: /etc/timezone
  notify: Reload timezone

This does mean that we need to create a few more directories and files for the template and handler.

$ mkdir -p /etc/ansible/roles/cumulus-base/templates
$ mkdir -p /etc/ansible/roles/cumulus-base/handlers

In the “templates” folder we’ll create “timezone.j2”, which will contain one variable which we’ll call “timezone”:

{{ timezone }}

We now need to tell Ansible what this variable should be set to. Because this is a widely applicable variable (not just applicable to an openstack lab) I’m going to put this into “/etc/ansible/group_vars/all/vars.yml” so that all hosts can reference it.
I’ll also add the NTP servers at the same time that we’ll reference in the next task below.

# NTP
timezone: "Pacific/Auckland"
ntp_servers:
  - "0.nz.pool.ntp.org"
  - "1.nz.pool.ntp.org"
  - "2.nz.pool.ntp.org"
  - "3.nz.pool.ntp.org"

In the “handlers” folder we’ll create “main.yml” and create a handler that tells the system to reload the timezone if a change is made.
On Debian, this is done by telling dpkg to reconfigure the “tzdata” package.

---
- name: Reload timezone
  command: "dpkg-reconfigure --frontend noninteractive tzdata"

Configuring NTP Servers

The task to configure NTP servers is going to look very similar to the timezone task, as we’re using the template module to modify an existing file and then run a handler if something changes.

- name: Configure NTP
  template:
    src: ntp.j2
    dest: /etc/ntp.conf
  notify: Restart NTP

For the template, I’m simply going to copy the existing configuration file, remove the extraneous comments, and then add in our NTP servers using a jinja ‘for’ loop.

# {{ ansible_managed }}
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

# NTP Servers
{% for item in ntp_servers %}
server {{ item }}
{% endfor %}

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Specify interfaces, don't listen on switch ports
interface listen {{ cumulus_management_interface }}

On the last line we’re telling the switch to listen for NTP only on the management interface.
We’re going to be refering to the management interface again (in later posts), so I’m going to define a variable so I only have to change this in one place if the need arises.

Because the variable is only relevant to the cumulus switches, I’m going to put this into a new file - “/etc/ansible/group_vars/openstack-cumulus/vars.yml”:

# Management
cumulus_management_interface: "eth0"

The handler looks a little simpler for this task though, as we can use the “service” module to restart NTP on the hosts.

- name: Restart NTP
  service:
    name: ntp
    state: restarted

Configuring Authentication

Authentication in a production environment should probably include incorporation into the existing authentication service (RADIUS/TACACS/LDAP), but because we’re running a lab I’ll leave it out for now.

However, Cumulus does include their own command-line-esque tool called “netd” which requires users to be given permission to run commands.
This is controlled in the configuration file “/etc/netd.conf”:

# Control which users/groups are allowed to run "add", "del",  
# "clear", "abort", and "commit" commands.  
users_with_edit = root, cumulus  
groups_with_edit = netedit

# Control which users/groups are allowed to run "show" commands.  
users_with_show = root, cumulus  
groups_with_show = netshow, netedit

Because I might want to play with this in the future, I’ll add the user “vagrant” into the “netedit” group using the user module in Ansible:

- name: Allow administrator access to NCLU (net commands)
  user:
    name: '{{ ansible_user }}'
    groups: netedit
    append: yes

Remember that the variable ansible_user refers to the user we’re using to log into the device with, which we set in an earlier post in the “hosts” inventory file.

Testing

Now that our role is all put together, let’s give this a test run to make sure that everything works as intended:

$ ansible-playbook /etc/ansible/playbooks/openstack-cumlus.yml
...
PLAY RECAP ****************************************************************
cumulus-leaf01             : ok=8    changed=7    unreachable=0    failed=0
cumulus-leaf02             : ok=8    changed=7    unreachable=0    failed=0
cumulus-leaf03             : ok=8    changed=7    unreachable=0    failed=0
cumulus-leaf04             : ok=8    changed=7    unreachable=0    failed=0
cumulus-spine01            : ok=8    changed=7    unreachable=0    failed=0
cumulus-spine02            : ok=8    changed=7    unreachable=0    failed=0

And one more run to make sure that nothing changes the second time:

$ ansible-playbook /etc/ansible/playbooks/openstack-cumlus.yml
...
PLAY RECAP ****************************************************************
cumulus-leaf01             : ok=6    changed=0    unreachable=0    failed=0
cumulus-leaf02             : ok=6    changed=0    unreachable=0    failed=0
cumulus-leaf03             : ok=6    changed=0    unreachable=0    failed=0
cumulus-leaf04             : ok=6    changed=0    unreachable=0    failed=0
cumulus-spine01            : ok=6    changed=0    unreachable=0    failed=0
cumulus-spine02            : ok=6    changed=0    unreachable=0    failed=0

Running with Vagrant

Now that we’ve got a playbook written we can incorporate this into Vagrant.
The aim is that this will be run when vagrant up is called, just like the ZTP playbook in the last post.
To recap, this simply requires the .vm.provision block somewhere in our Vagrantfile.

However, unlike our previous post, we are running this playbook against multiple guest machines instead of just the one, so we end up with two options to have the playbook run:

Include the .vm.provision block each individual Cumulus guest block, and use the limit option that the provisioner provides. This would run the playbook once for every machine specified and only against that one machine.
Include the .vm.provision block only once, in the last Cumulus guest block.

I’m going to go with the second option as I believe this to be a cleaner way of completing this.
However, I can see that you might want to run this individually if there was a requirement to have one device fully configured before bringing up the next.

This is what the Vagrantfile ends up looking like. I’ve included a skeleton of the last Cumulus machine’s configuration to illustrate the provisioners’ position in the configuration hierachy.

# Leaf 4
config.vm.define "cumulus_leaf04" do |device|
  ...
  # Ansible Playbook for all Cumulus switches
  device.vm.provision :ansible do |ansible|
    ansible.playbook = "/etc/ansible/playbooks/openstack-cumulus.yml"
    ansible.inventory_path = "/etc/ansible/hosts"
    ansible.limit = "openstack_cumulus"
  end
end

The only thing to note here is that we also need to use ansible.limit to specify all of the hosts we want to run against, as otherwise Vagrant will tell Ansible to only run this against the hosts which are in the same block; which in this case would be just “cumulus_leaf04”.

Conclusion

In this post I started an Ansible playbook to configure some basic functions on Cumulus hosts and covered how to include this in our Vagrantfile.

In the next post, I’ll dive into the network interface configuration of the switches, including LLDP and PTMD (I’ll cover what PTMD is as well) configuration on Cumulus.

Appendices

References

Ansible - “apt” module
Ansible - “hostname” module
Ansible - “template” module
Ansible - “command” module
Ansible - “service” module
Ansible - “user” module
Vagrant - Ansible Provisioner
Cumulus Linux - Setting the hostname
Cumulus Linux - Setting Date and Time
Cumulus Linux - NCLU, Editing netd.conf

Versions used

Desktop Machine: kubuntu-18.04
VirtualBox: virtualbox-5.2.10
Vagrant: 2.2.1
Cumulus VX Vagrant Box: CumulusCommunity/cumulus-vx (virtualbox, 3.7.2)
Ansible: 2.7.2

Ansible - Firewalld Module on Ubuntu

Sat, 17 Nov 2018 00:00:00 +0000

Overview

When trying to run an Ansible “firewalld” task on an Ubuntu machine recently, I received the following error:

TASK [Firewalld Task] **************************
fatal: [vagrant-test]: FAILED! => {"changed": false, "msg": "Python Module not found: firewalld and its python module are required for this module, version 0.2.11 or newer required (0.3.9 or newer for offline operations)"}

Here is the task that I’m running:

name: Firewalld Test Task
firewalld:
  service: https
  state: enabled

Investigation

After a quick google search, I found some other people have run into this as well.
Although there were some fixes noted in the above issues, none of them sat well with me as they required installing extra packages.

There has to be a better way right?

Hunting further took me back to the Ansible firewalld module documentation, which includes the following notes:

Not tested on any Debian based system.

Requires the python2 bindings of firewalld, which may not be installed by default.

For distributions where the python2 firewalld bindings are unavailable (e.g Fedora 28 and later) you will have to set the ansible_python_interpreter for these hosts to the python3 interpreter path and install the python3 bindings.

Those last points ring true with us. But how do we go about fixing this?

Rectification

Well, turns out that the fix is considerably easier than installing extra packages.

Following the advice from that last note in the documentation, we first need to check what version of the bindings we’re using:

$ apt-cache depends firewalld | grep Depends | grep python
  Depends: python3-dbus
  Depends: python3-gi
  Depends: python3-slip-dbus
  Depends: <python3:any>

Now that we’ve confirmed we’re using the python3 bindings, we’ll simply tell Ansible to use python3 when running the task.
This is accomplished by setting the variable ansible_python_interpreter to “/usr/bin/python3”.

In true Ansible fashion, the variable can be set in multiple ways/places. But because I don’t want it to impact the running of my other tasks, I’m going to use the task level vars option, as below:

name: Firewalld Test Task
firewalld:
  service: https
  state: enabled
vars:
  ansible_python_interpreter: '/usr/bin/python3'

This results in a happy firewalld task and a change of the firewalld state on the host:

TASK [Firewalld Task] **************************
changed: [vagrant-test]

Conclusion

This was another quick post, which may help other people who are running into a similar issue.

I could have gone into more detail around how I might refactor this task to include logic so that the python interpreter is only changed when running against Ubuntu machines, but I wanted to keep this quick and easy to read.

Appendices

References

Ansible Github Issue #24855
Ansible Modules Extras Github Issue #1282

Versions used

Local Machine:

Operating System: Kubuntu-18.04.1
Ansible: 2.7.2

Remote Machine:

Operating System: Ubuntu-16.04.5
Firewalld: 0.4.0

Python Virtualenv for Dummies

Sun, 05 Aug 2018 00:00:00 +0000

Overview

Recently I was trying to write a script that had dependencies on pip packages. Now, I am fortunate enough to know that it’s a bad idea to install pip packages globally (especially using sudo), so I set about to find the best way to install those dependencies without breaking my system.

I’d heard about virtualenv (Virtual Environment) before, and after a quick google this seemed to be the solution I needed.
Simply put, a virtualenv is a secluded environment (directory) where you install the dependencies your application requires so that it can’t interfere with the rest of your operating system (and its’ possible separate dependencies).

However, installing and using virtualenv seemed far too complicated, especially for a python newbie like myself.

This post is meant as a “Python Virtualenv for Dummies”; both so that I have a quick document to refer back to, and hopefully to help those as helpless as I am.

Note that this post uses an installation of Ubuntu 18.04 with Python v3.6. If you were wanting to install Virtualenv for Python 2 instead, simply remove the “3” off the end of the install commands.

Installation

First up is installing python and pip, both of which are done through apt:

$ sudo apt-get install python3 python3-pip

Next is installing virtualenv using pip. We’ll install this only for our user, so we don’t need to use sudo:

$ pip3 install --user virtualenv

Using Virtualenv

First we need to create an environment, this is done in the root directory of your project:

$ cd /path/to/project/dir
$ virtualenv venv
New python executable in venv/bin/python
Installing setuptools, pip, wheel...done.

The only argument supplied to the command is the subdirectory to create within the directory.
The name “venv” is a common one, but you can name it whatever you want.

Don’t forget to add this directory to your .gitignore (or equivalent) if you’re using SCM.

Within this newly created directory are the python binaries that we can use to only affect our project. For example, to install pip packages to our environment:

venv/bin/pip install $PackageName

Or to run a python program:

venv/bin/python $ProgramName.py

Activating the environment

If you’re like me and are too lazy to type in the path to the binaries folder every time you want to run a command, then there is the option to activate the environment.

$ source venv/bin/activate

This will change your shell prompt to show you that you’re running python commands from within a venv as well.

Now we can run commands without referencing the binaries folder:

$ pip install $PackageName

Behind the scenes, all this does is change the $PATH entry for the current shell session so that the first entry is the local virtualenv.

$ $PATH
bash: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:
$ source venv/bin/activate
$ $PATH
bash: /path/to/project/dir/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:

To reverse this, either leave the current shell session or run the deactivate script:

$ deactivate

Conclusion

Hopefully this post was short and simple enough to understand, as I expect to refer back to this myself once I inevitably forget how to use virtualenv in a month.

Appendices

References

Virtualenv Official Documentation
A non-magical introduction to Pip and Virtualenv for Python beginners

Versions used

Desktop Machine: kubuntu-18.04.1
Python: Python 2.7.15rc1
Virtualenv: 16.0.0