Oh, that was really easy…

I just bought an Apple Magic Keyboard. My initial reaction is awesome. This is because of the ease of pairing with another Apple device. To pair this you literally turn it on and then plug it into your computer with the supplied lightning cable. No passcodes, no discovery mode, just plug it in and it works. Given that Bluetooth and USB go hand in hand these days, I really think that nearly anything that requires bluetooth pairing should work this way.

Why buy the keyboard?  I’m one of  the many software developer/devops engineer/sysadmin guys who’s avoided upgrading to the latest generation of Apple laptop mainly because of the new keyboards:

  • Forcing me to use the touch bar for the Esc key is honestly a complete non-starter.
  • and, the reduced travel of the butterfly keyboard, combined with the fact that if you get a crumb in it you need to take it back to apple to get it repaired. This is another non-starter.

So, I’ve been slogging through life with the top of the line 2015 15″ MacBook Pro for quite a few years. To pull me over the hump, a new MacBook Pro would have to be:

  • Quad-Core i7 or better
  • 32GB of RAM
  • 15″ Display

If such a machine had the keyboard from the the 2015 MacBook Pro, I would have already bought it.

But my current laptop is starting to show it’s age. I have to recondition the battery before a long flight to maximize battery lifetime. The current machine’s dusty enough inside that the fans have lost some of their efficiency.

For $99.00, and even less from Amazon, I can try out the new mechanism and make a better evaluation of my ability to use the new laptop. I’m typing this blog post with the new machine and I have to admit that the new mechanism is nice. And, in the worst case, this would always be a good media center PC keyboard.

Ansible step zero

In my previous article I showed the steps to take to build an ansible repository that you could grow to fit your existing infrastructure. The first step here to setup the repository that you built to self-bootstrap. For that you’ll need to flesh out your inventory and build your first playbook.

Building Inventory

Ansible is driven off of an inventory. The inventory specifies the elements of your infrastructure and the groups them. This is to make things easy to manage. Ansible is compatible with three kinds of inventory: Inventory specified as a Windows style .ini formatted static file, or specified in a yaml file, or finally specified dynamically. Dynamic inventory is the holy grail. I recommend starting with a yaml inventory.

Although both yaml and ini style inventories have roughly the same capabilities, I prefer yaml because if you work with ansible, you’re going to become good friends with yaml no matter what. If you aren’t familiar with yaml format, find some time to study it. yaml is just a markup format that allows you to structure things. I didn’t really get yaml until I played with the python yaml module. I realized that yaml, like json, allows you to write python variables into a file in a structured fashion. the python yaml module can read a properly formatted yaml file and will return a python variable containing the contents of the yaml “document” or it can take any python variable, an array, a dict, a static, and write it such that another python program could read it. Yaml differs from json in that it’s generally parseable and readable by human beings. If the consumer of your data is program, use json. If a human is expected to read it, use yaml.

Your starting yaml inventory should look something like this:

          - ansible
          - terraform
          - git
          - emacs

          my_domain: mydomain.com
          my_host: maestro-test

This defines an inventory with one group: maestro-test. It includes one machine at IP address and it defines some variables for the group. This should be stored in an approriately named file:


In the Ansible directory.

The first playbook

With an inventory, you can build a playbook. The first playbook looks like this:

- hosts: maestro-test
- tasks:
    - name: Install standard packages
        name: "{{ item }}"
        state: latest
      with_items: "{{ std_pkg }}"

    - name: Install additional packages
        name: "{{ item }}"
        state: latest
      with_items: "{{ add_pkg }}"

This should be installed in a file named something like:


in the Ansible directory. At this point presuming that you have a machine, physical or virtual at into which you can ssh, as root, you can bootstrap your maestro as follows:

chris $ ansible-playbook -i base-maestro-inventory.yml --user root base-maestro-playbook.yml

And that should install the correct packages onto your maestro test box. I’ll revisit this article later to add users.

Getting started with Ansible, et al

For admins, young and old, getting started with orchestration tools like ansible I believe that the wise man’s first move is to create an orchestration workstation. This machine will have: ansible, terraform, git, and  your favorite editor. You are going to use this machine as the basis for infrastructure as code for your organization for the short term future. Basically, you’ll stop using this machine for infrastructure as code once you get to the point where you can repeatably automate the creation and change management of things. At that point the role of this machine will be testing infrastructure changes. And there will be another machine exactly like this one that controls your production infrastructure.

The first thing that this machine should be able to do is replicate itself. That’s a simple task. In Unix terms you are looking at a box that can:

  • allows you to log in via ssh keys
  • allows you to edit the ansible and terraform configurations which
  • are stored in git so that they are version controlled

That really specifies three users, you, ansible, and terraform. Also, as specified before, you need a hand full of packages: ansible, git, and your favorite editor. The whole thing looks pretty similar to this:

chris $ mkdir Ansible
chris $ git init Ansible
chris $ cd Ansible
chris $ mkdir -p files/global group_vars host_vars roles/dot.template/{defaults,files,handlers,tasks,templates,tests}
chris $ find * -type d -exec touch {}/Readme.md \;
chris $ touch Readme.md
chris $ git add . && git commit -m 'Initial revision.'

That builds an ansible configuration as a git repository and checks in the first revision. It also populates the ansible repository with directories that  roughly correspond to ansible best practices. This will be a working repository which you are going to build out to support your infrastructure. You’ll do this by adding inventory, playbooks and roles bespoke to your needs.

More on this later.

Setting up OS X Mail

Old hand Unix guys like myself are used to plain text email. OS X mail will gladly oblige me if I go to: Preferences –> Composing –> Message Format: { Plain Text }. But Apple chooses to use a proportionally spaced font: Helvetica by default. To fix this, go to: Preferences –> Fonts & Colors –> Use fixed width font for plain text messages. I think that these two should go hand in hand but Apple thinks otherwise.

Sadly, what you cannot fix is the line width. It would be nice to have the plain text message lines wrapped at the nth column but, I’ll take what I can get.

Changing VMware Fusion network settings

For those that run VMware Fusion, the “/Library/Preferences/VMware Fusion”  directory on the Mac is a wealth of information.

$ cd /Library/Preferences/VMware\ Fusion/
$ ls -l
total 40
-r--r--r-- 1 root wheel 31 Nov 17 11:01 lastLocationUsed
-rw-r--r-- 1 root wheel 548 May 5 2018 license-fusion-100-e3-201704
-rw-r--r-- 1 root wheel 689 May 5 2018 license-fusion-100-e4-201704
-rw-r--r-- 1 root wheel 547 Dec 6 2013 license-fusion-50-e3-201202
-rw-r--r-- 1 root wheel 547 Apr 10 2014 license-fusion-60-e3-201303
-rw-r--r-- 1 root wheel 547 Oct 31 2014 license-fusion-70-e3-201404
-rw-r--r-- 1 root wheel 688 Oct 25 2014 license-fusion-70-e4-201404
-rw-r--r-- 1 root wheel 547 Jun 23 2016 license-fusion-80-e3-201505
-rw-r--r-- 1 root wheel 740 Nov 3 05:54 networking
-rw-r--r-- 1 root wheel 740 Aug 7 20:23 networking.bak.0
drwxr-xr-x 10 root wheel 340 Nov 17 11:01 thnuclnt
drwxr-xr-x 4 root wheel 136 Dec 6 2013 vmnet1
drwxr-xr-x 7 root wheel 238 Dec 6 2013 vmnet8

The license-fusion… files have your license keys as well as other information in them. But today, the jewel for me is the networking file:

$ cat networking
answer VNET_1_DHCP yes
answer VNET_1_DHCP_CFG_HASH E08B... ...D0D8
answer VNET_1_HOSTONLY_SUBNET 172.a.b.0
answer VNET_8_DHCP yes
answer VNET_8_DHCP_CFG_HASH 2031... ...F498
answer VNET_8_HOSTONLY_SUBNET 10.c.d.0
answer VNET_8_NAT yes

This file defines the networks that your host-only and nat network adapters use. VMware appears smart enough to avoid network collisions e.g. using for the NAT adapter at vmnet8 when that’s also the network configured on your home router.

According to this article (VMware login required), simply editing this file and restarting VMware Fusion’s networking component should change the dhcp setting that your machine uses. Any skilled system or network administrator should be able to get their hands around that.

Finally, the lines that specify DHCP hashes appear to be the mechanism that VMware uses to detect changes in the networking file. If you dig deeper, there’s a directory for vmnet1.

$ ls -l vmnet1
total 8
-rw-r--r--  1 root  wheel  1575 Nov 17 11:01 dhcpd.conf
-rw-r--r--  1 root  wheel  1575 Nov 17 11:01 dhcpd.conf.bak
$ cat vmnet1/dhcpd.conf
# Configuration file for ISC 2.0 vmnet-dhcpd operating on vmnet1.
# This file was automatically generated by the VMware configuration program.
# See Instructions below if you want to modify it.
# We set domain-name-servers to make some DHCP clients happy
# (dhclient as configured in SuSE, TurboLinux, etc.).
# We also supply a domain name to make pump (Red Hat 6.x) happy.

###### VMNET DHCP Configuration. Start of "DO NOT MODIFY SECTION" #####
# Modification Instructions: This section of the configuration file contains
# information generated by the configuration program. Do not modify this
# section.
# You are free to modify everything else. Also, this section must start
# on a new line
# This file will get backed up with a different name in the same directory
# if this section is edited and you try to configure DHCP again.

# Written at: 11/17/2018 11:01:21
allow unknown-clients;
default-lease-time 1800; # default is 30 minutes
max-lease-time 7200; # default is 2 hours

subnet 172.a.b.0 netmask {
range 172.a.b.128 172.a.b.254;
option broadcast-address 172.a.b.255;
option domain-name-servers 172.a.b.1;
option domain-name localdomain;
default-lease-time 1800; # default is 30 minutes
max-lease-time 7200; # default is 2 hours
host vmnet1 {
hardware ethernet 00:50:56:x:y:z;
fixed-address 172.a.b.1;
option domain-name-servers;
option domain-name "";
####### VMNET DHCP Configuration. End of "DO NOT MODIFY SECTION" #######


This is just a standard dhcpd.conf file as you would see if you ran isc-dhcpd. The interesting thing is that the hash is what you get if you do this:
$ sed -ne '/VMNET DHCP.*Start/,/VMNET DHCP.*End/ p' vmnet1/dhcpd.conf | shasum
e08b... ...d0d8 -

The more you know…

Testing python scripts

You have a python script that has some unit tests that you want to run:

python -c "m = __import__("my-script"); m.unittest()"

Runs the unittest() function within your script. N.B. this still works if your script has a hypen in the name which is my style.

Mutt account passwords

First, to give credit where it’s due, I started here. That said, here’s how I store and access account passwords in mutt on Linux.

## -- Passwords: encrypted by gpg --------------------------------------------------------------

source “/bin/gpg -d ~/.keychain/mutt.password.neopost.gpg 2>/dev/null |”

The source line in gpg tells mutt to decrypt a file at startup. The file .keychain/mutt… contains two mutt configuration lines:

set imap_pass = "<my_email_password>"
set smtp_pass = "<my_email_password>"

I created it as follows:

$ cat <<EOF | gpg -r <my_gpg_id> ~/.keychain/mutt.password.neopost.gpg
set imap_pass = "<my_email_password>"
set smtp_pass = "<my_email_password>"

Gpg knows how to decrypt this file and retrieve the plain text configuration. Note well that I used a “Here” document to create the file. This keeps mail password out of the filesystem. Simple stuff, at mutt startup the first time I use it, gpg-agent asks for my gpg key and unlocks the configuration snippet.

Of IPv6, and unhappy eyeballs

I’m a dinosaur. I still bring a little Soekris Net5501 configured as a NAT gateway with IPsec back home when on vacation. This solution works best in hotels that still offer Wired internet. I do this because:

  • Majorly, In many cases a hotel’s WiFi is a great example of the worst case engineering solution with wifi tuned just well enough that web browsing doesn’t suck but nothing else is considered.
  • Minorly, Hotel’s in the United States will probably be the last places on Earth to get IPv6.

The other advantages should be obvious. By bringing a router I can control or at least police my DNS.

My Soekris is tiny. It takes up about as much space as three tee shirts in my packed bag including an Ancient Apple Airport Express that can be tuned to run on 5GHz only. Off hand I want to mention that configuring 5GHz WiFi in each hotel room might bring the best improvement to WiFi overall but hoteliers can’t do this because lots of people have old devices that don’t do 5GHz or don’t do it well. No one wants to be the guy trying to explain to a customer that the problem isn’t the WiFi, it’s the customer’s 2007 vintage MacBook Pro that can’t use channel 149.

So, note my surprise when I find that things aren’t working better on my custom wifi, homed to the edge router over ethernet, during my latest hotel stay. What happened? I think this…

My router connects the internet and establishes a tuned IPSec tunnel back home. Once that works, it advertises IPv6 addresses from my /48 which route back to my house and then out to the internet. If the latency for all of that is 10ms no one would notice the difference. If it goes greater than 30ms things start getting funky. We’ve crossed into a time where if you have a dual stack machine, one with both IPv4 and IPv6, you’ll try to use IPv6 for outbound work first and revert to IPv4 only when v6 has too much latency. Well, folks, when you have just enough latency, this doesn’t look pretty.

No solution yet and possibly not ever. The problem with latency issues like this is that you can’t easily fix them. That’s why tuning latency out is so important. I’ll post more when I know more.

Emacs use tabs rather than spaces.

Today, about the only place you should see an ascii TAB in a file is in a Makefile. In a world where memory is metered as gigabytes of RAM and terabytes of storage on fast SSDs there is absolutely no need to save space in a source code or configuration file by using a tab rather than two or four or eight spaces. Note well that I may be talking to your editor configuration and not you. But when you write code you should say what you mean and mean what you say unequivocally. I say this because I have been looking at a whitespace difference in my puppet checks for better than a month now. This is because my file has in production has tabs and my file in the puppet/git repository has spaces.

To that end, I’m linking this bit of Emacs magic for readers and my future self.