Ansible step zero

In my previous article I showed the steps to take to build an ansible repository that you could grow to fit your existing infrastructure. The first step here to setup the repository that you built to self-bootstrap. For that you’ll need to flesh out your inventory and build your first playbook.

Building Inventory

Ansible is driven off of an inventory. The inventory specifies the elements of your infrastructure and the groups them. This is to make things easy to manage. Ansible is compatible with three kinds of inventory: Inventory specified as a Windows style .ini formatted static file, or specified in a yaml file, or finally specified dynamically. Dynamic inventory is the holy grail. I recommend starting with a yaml inventory.

Although both yaml and ini style inventories have roughly the same capabilities, I prefer yaml because if you work with ansible, you’re going to become good friends with yaml no matter what. If you aren’t familiar with yaml format, find some time to study it. yaml is just a markup format that allows you to structure things. I didn’t really get yaml until I played with the python yaml module. I realized that yaml, like json, allows you to write python variables into a file in a structured fashion. the python yaml module can read a properly formatted yaml file and will return a python variable containing the contents of the yaml “document” or it can take any python variable, an array, a dict, a static, and write it such that another python program could read it. Yaml differs from json in that it’s generally parseable and readable by human beings. If the consumer of your data is program, use json. If a human is expected to read it, use yaml.

Your starting yaml inventory should look something like this:

          - ansible
          - terraform
          - git
          - emacs

          my_host: maestro-test

This defines an inventory with one group: maestro-test. It includes one machine at IP address and it defines some variables for the group. This should be stored in an approriately named file:


In the Ansible directory.

The first playbook

With an inventory, you can build a playbook. The first playbook looks like this:

- hosts: maestro-test
- tasks:
    - name: Install standard packages
        name: "{{ item }}"
        state: latest
      with_items: "{{ std_pkg }}"

    - name: Install additional packages
        name: "{{ item }}"
        state: latest
      with_items: "{{ add_pkg }}"

This should be installed in a file named something like:


in the Ansible directory. At this point presuming that you have a machine, physical or virtual at into which you can ssh, as root, you can bootstrap your maestro as follows:

chris $ ansible-playbook -i base-maestro-inventory.yml --user root base-maestro-playbook.yml

And that should install the correct packages onto your maestro test box. I’ll revisit this article later to add users.

Getting started with Ansible, et al

For admins, young and old, getting started with orchestration tools like ansible I believe that the wise man’s first move is to create an orchestration workstation. This machine will have: ansible, terraform, git, and  your favorite editor. You are going to use this machine as the basis for infrastructure as code for your organization for the short term future. Basically, you’ll stop using this machine for infrastructure as code once you get to the point where you can repeatably automate the creation and change management of things. At that point the role of this machine will be testing infrastructure changes. And there will be another machine exactly like this one that controls your production infrastructure.

The first thing that this machine should be able to do is replicate itself. That’s a simple task. In Unix terms you are looking at a box that can:

  • allows you to log in via ssh keys
  • allows you to edit the ansible and terraform configurations which
  • are stored in git so that they are version controlled

That really specifies three users, you, ansible, and terraform. Also, as specified before, you need a hand full of packages: ansible, git, and your favorite editor. The whole thing looks pretty similar to this:

chris $ mkdir Ansible
chris $ git init Ansible
chris $ cd Ansible
chris $ mkdir -p files/global group_vars host_vars roles/dot.template/{defaults,files,handlers,tasks,templates,tests}
chris $ find * -type d -exec touch {}/ \;
chris $ touch
chris $ git add . && git commit -m 'Initial revision.'

That builds an ansible configuration as a git repository and checks in the first revision. It also populates the ansible repository with directories that  roughly correspond to ansible best practices. This will be a working repository which you are going to build out to support your infrastructure. You’ll do this by adding inventory, playbooks and roles bespoke to your needs.

More on this later.

Changing VMware Fusion network settings

For those that run VMware Fusion, the “/Library/Preferences/VMware Fusion”  directory on the Mac is a wealth of information.

$ cd /Library/Preferences/VMware\ Fusion/
$ ls -l
total 40
-r--r--r-- 1 root wheel 31 Nov 17 11:01 lastLocationUsed
-rw-r--r-- 1 root wheel 548 May 5 2018 license-fusion-100-e3-201704
-rw-r--r-- 1 root wheel 689 May 5 2018 license-fusion-100-e4-201704
-rw-r--r-- 1 root wheel 547 Dec 6 2013 license-fusion-50-e3-201202
-rw-r--r-- 1 root wheel 547 Apr 10 2014 license-fusion-60-e3-201303
-rw-r--r-- 1 root wheel 547 Oct 31 2014 license-fusion-70-e3-201404
-rw-r--r-- 1 root wheel 688 Oct 25 2014 license-fusion-70-e4-201404
-rw-r--r-- 1 root wheel 547 Jun 23 2016 license-fusion-80-e3-201505
-rw-r--r-- 1 root wheel 740 Nov 3 05:54 networking
-rw-r--r-- 1 root wheel 740 Aug 7 20:23 networking.bak.0
drwxr-xr-x 10 root wheel 340 Nov 17 11:01 thnuclnt
drwxr-xr-x 4 root wheel 136 Dec 6 2013 vmnet1
drwxr-xr-x 7 root wheel 238 Dec 6 2013 vmnet8

The license-fusion… files have your license keys as well as other information in them. But today, the jewel for me is the networking file:

$ cat networking
answer VNET_1_DHCP yes
answer VNET_1_DHCP_CFG_HASH E08B... ...D0D8
answer VNET_1_HOSTONLY_SUBNET 172.a.b.0
answer VNET_8_DHCP yes
answer VNET_8_DHCP_CFG_HASH 2031... ...F498
answer VNET_8_HOSTONLY_SUBNET 10.c.d.0
answer VNET_8_NAT yes

This file defines the networks that your host-only and nat network adapters use. VMware appears smart enough to avoid network collisions e.g. using for the NAT adapter at vmnet8 when that’s also the network configured on your home router.

According to this article (VMware login required), simply editing this file and restarting VMware Fusion’s networking component should change the dhcp setting that your machine uses. Any skilled system or network administrator should be able to get their hands around that.

Finally, the lines that specify DHCP hashes appear to be the mechanism that VMware uses to detect changes in the networking file. If you dig deeper, there’s a directory for vmnet1.

$ ls -l vmnet1
total 8
-rw-r--r--  1 root  wheel  1575 Nov 17 11:01 dhcpd.conf
-rw-r--r--  1 root  wheel  1575 Nov 17 11:01 dhcpd.conf.bak
$ cat vmnet1/dhcpd.conf
# Configuration file for ISC 2.0 vmnet-dhcpd operating on vmnet1.
# This file was automatically generated by the VMware configuration program.
# See Instructions below if you want to modify it.
# We set domain-name-servers to make some DHCP clients happy
# (dhclient as configured in SuSE, TurboLinux, etc.).
# We also supply a domain name to make pump (Red Hat 6.x) happy.

###### VMNET DHCP Configuration. Start of "DO NOT MODIFY SECTION" #####
# Modification Instructions: This section of the configuration file contains
# information generated by the configuration program. Do not modify this
# section.
# You are free to modify everything else. Also, this section must start
# on a new line
# This file will get backed up with a different name in the same directory
# if this section is edited and you try to configure DHCP again.

# Written at: 11/17/2018 11:01:21
allow unknown-clients;
default-lease-time 1800; # default is 30 minutes
max-lease-time 7200; # default is 2 hours

subnet 172.a.b.0 netmask {
range 172.a.b.128 172.a.b.254;
option broadcast-address 172.a.b.255;
option domain-name-servers 172.a.b.1;
option domain-name localdomain;
default-lease-time 1800; # default is 30 minutes
max-lease-time 7200; # default is 2 hours
host vmnet1 {
hardware ethernet 00:50:56:x:y:z;
fixed-address 172.a.b.1;
option domain-name-servers;
option domain-name "";
####### VMNET DHCP Configuration. End of "DO NOT MODIFY SECTION" #######


This is just a standard dhcpd.conf file as you would see if you ran isc-dhcpd. The interesting thing is that the hash is what you get if you do this:
$ sed -ne '/VMNET DHCP.*Start/,/VMNET DHCP.*End/ p' vmnet1/dhcpd.conf | shasum
e08b... ...d0d8 -

The more you know…

More WiFi isn’t always better WiFi

I generally like my Cable TV company. This is mostly because they have realized that they won’t be a Cable TV company in ten years and they are running their business according to that mantra. In English, this means that they are putting their effort in being a very very good consumer grade ISP. One of the side benefits that Cable Internet has been offering is access to public hotspots. As a consortium these companies have nearly every urban area that I’ve been to covered in WiFi. On the surface this seems to be a good thing but it has some downsides. Most WiFi uses the 2.4GHz frequency band. WiFi routers ship with a range in open air of about 450m. There are only three non-overlapping channels in the 2.4GHz band. If you’ve debugged WiFi in an urban setting, you are probably painfully aware of these three characteristics of 2.4GHz WiFi. When they Cable TV company starts deploying open hotspots in your neighborhood, they aren’t pushing the system to a solution for this problem.

I have to qualify this as a rant though since there isn’t much that anyone can do to fix the problem. Especially since the Cable TV companies have started enhancing their WiFi offering by giving away high powered WiFi routers which that offer dual SSID to their customers. It won’t be long before anyone who want’s to do anything will be in 5GHz.

Apple’s Captive Network Assistant

In an attempt to make life easier Apple added the Captive Network Assistant App to OS X. I think this addition was made sometime around Lion. Captive Network Assistant is an App that can display a little the very simple web page you get when you connect to a wifi network that has Captive Portal. These are the pages you get when you first log onto your coffee shop WiFi. They usually ask you to agree to some terms and conditions before you can use the network. In the case of hotels, resorts, and cruise ships they will also tie to the site billing system so you can be charged if that’s appropriate. Lately I’ve started to get these sites on both my MiFi hotspot and most lately, my home WiFi. This article explains three major drawbacks to Apple’s approach here. The authors of these web pages will frequently embed logout information into the page when the captive portal mechanism is being used to track usage for billing. In this use case, the app is a hinderance because when it disappears, it takes the logout link with it. Also, Apple triggers the app by attempting to fetch a known page over the web when your WiFi first connects. If it doesn’t get what it expects, it knows it’s behind a Captive Portal. In my case, the Captive Portal App is displaying Apple’s static page which indicates that you aren’t behind a portal.

When I started seeing captive portal on my home network, I decided the turn the thing off. To turn captive portal off do this command:

sudo defaults write /Library/Preferences/SystemConfiguration/ Active -boolean false

To restore the old behavior, do this, again in a terminal window:

sudo defaults delete /Library/Preferences/SystemConfiguration/ Active

Other people including the article linked above recommend renaming the App. I’m not in love with that solution, mostly because two months from now I don’t expect to remember that I did this in the first place. My solution isn’t much better. One could argue it’s worse because it requires terminal and sudo. It’s the one I went with though.

NFS – Old habits die hard

Old habits and myths die hard. Conventional wisdom asserts that UDP is better because it has lower overhead; then conventional wisdom suggests that you tune the buffer sizes to improve performance. On the face of things that would seem to work but once the the write size exceeds the max packet size, NFS delivers the packet by using multiple packets. Sending multiple packets triggers the issue because dropping just one UDP packet means the whole buffer must be resent. Contrast with TCP: yes the packet header is larger so less data can be sent and yes the receiving side has to ack each packet. But: with TCP if a packet gets dropped, only that packet needs to be resent; with a modern TCP stack the kernel will constantly adjust the window size to make the best use the available bandwidth. In other words NFS over tcp will automatically tune the buffer sizes for the current conditions.

Website setup.

This seems obvious but it’s really handy to have my website setup with DAV access to the backend for administration purposes. I’ve recently setup on of my sites this way and it works out quite nicely.

bogofilter and the corrupt wordlist.db

Looks like something burped on my mailserver and my bogofilter wordlist got too big. Probably something to do with limits anyhow. In any case I was looking for a way to recover from the issue and came across this pearl in the Bogofilter FAQ. Well, the advice is incomplete. If you really hose up the database then bogoutil -d will stop printing  entries before the end of the database. The next recovery step is to use the db utilities: db_dump and db_load to fix the database. db_dump -r (on FreeBSD db_dump-<version>) dumps the database into a text file and db_load creates a text file from a word list. The problem is that the advice in the bogofilter faq is out of date. It looks like there are some parameters that have to be specified. My solution: use db_dump without the -r that creates a broken database with a default header. Copy the header into the new text file and then append the output of db_dump -r to that. Et voila!

Alright, it’s no longer 1998!

One thing that really ticks me off the web designer conversation where your web design guy insists on designing to an 800×600 screen resolution to ensure that your pages will be accessible by everyone on the web. Today I ran across this nugget (opens in a new window). I’ve always said that this is so 1998 yet I’ve had this conversation as recently as 2007. Well, if you dissect the table you come up with this:

1920×1200 2.27%
1680×1050 8.72%
1440×900 18.37%
1366×768 20.76%
1280×1024 —-
1280×800 —-
1280×768 58.09%
1152×864 61.04%
1024×768 94.94%
800×600 100.00%
1920×1200 2.27%
1680×1050 8.72%
1280×1024 21.97%
1440×900 31.62%
1152×864 34.57%
1280×800 56.92%
1366×768 —-
1280×768 —-
1024×768 94.94%
800×600 100.00%

That’s right. If you design for 1024×768 you reaching nearly 95% of all the web browsers that participated in this survey. Now web designers can partly like it’s 2004!