If anybody ever says that Apple is a lot better than Microsoft one thing that they need to pay attention to is the fact that both companies are guilty of the same problems. In this case I’m talking about both companies habit of letting long standing regressions in their software languish, unaddressed for long periods of time. Apple’s sin in this case is with Disk Utility. Apples has allowed a bug in the Disk Utility in macOS go on, unaddresses since about OS X 10.13 or so when they changed the structure of Time Machine Backups to force an encrypted drive. I’ll admit that I’m not being completely fair. I’m running an older version of macOS on my laptop so this bug may indeed be fixed but it still stayed in the software for a good 3 years.
By trying to make it easier to use an encrypted volume for backups Apple has added a few steps to the process of checking these volumes for structural errors. This means the graphic Disk Utility frequently false positives, saying that your volume has a problem. The real issue is that Disk Utility hasn’t properly set things up for the volume check to happen. Back in the olden days, UNIX you wouldn’t let you use a Volume with structural problems because you couldn’t mount it with write allowed. Today is you can mount broken drives in write mode. Then you get to cross your fingers that you’re not compounding an existing problem. Side note: Here’s where I admit to being really really old because 99% of the time its actually okay and that’s actually the case. The result is that Disk Utility can’t properly check out your Time Machine Volumes. To check one out you need to take the time to boot your machine into recovery mode where all of this shiny that makes users happy is disabled. In recovery mode, Disk Utility just works. Compounding this problem, when Apple does the check from a normal boot, it doesn’t detect its own bug and declares that your volume is dangerously corrupted and unreliable so your best best is to start from scratch. This article shows how you can at least get some peace of mind by checking the state of the volume and repairing it from the command line in a terminal window. I would’ve liked to have seen a screenshot of the command line session. But the author decided that figuring out which disk you need to check is too difficult and they didn’t include one. That’s the responsible choice since you are going to be doing a lot of potentially destructive commands with sudo. I worked my way through the process on my own third Time Machine Volume. I have this issue because this Volume is connected to my docking station. It auto mounts when I use my machine on my desktop so I can have a full sized monitor. It’s easy to forget that the Volume needs to be ejected cleanly and quiesced before I disconnect from the docking station. I’m cultivating the habit of ejecting this Volume when my backup has completed.
Not many people chose to run their own gitlabs instance these days. My preference for self reliance means that I do. If you value self reliance I have recommendations:
- Use ansible, chef, or puppet to build your gitlab instance because you are going to build two.
- Build one gitlabs server for your groups consumption. Put this one in a data center close to your user for good performance.
- Build a second gitlabs server in a remote location, perhaps at your favorite cloud provider. Where ever the second gitlabs instance is, you’ll want either one way or bidirection access via https or ssh between the two servers.
- Follow the directions in your gitlabs: Help -> User Documentation -> Mirror a Repository. To mirror the repository from the primary to the secondary.
At this point, you’ve created a great plan B for disaster recovery in case something terrible happens to your gitlabs. For me, gitlabs is storing the Terraform and Ansible that I use to build my infrastructure. The goal is to be able to jumpstart your whatever from the mirror. I called the mirror my plan be because my plan A is to directly restore gitlabs from a nightly backup.
Setting up the mirror
Setting up the mirror is well documented. In broad strokes, here are the steps:
- On the mirror, create a group and project to hold the mirrored repository.
- Choose Push or Pull mirroring. In Push, the primary will push updates to the secondary as you work. In Pull, the mirror will periodically poll the primary for changes. You’ll have to decide what works best for you.
- Fill out the form and perform any needed setup. When using push over ssh, this means setting up the primary to push and then copying the ssh public key from the primary and adding it as an allowed key on the mirror user.
As you configure mirroring, remember that constructing the mirror URL can be tricky especially if you want to use ssh as transport. This is because a typical git cloning string looks like this: firstname.lastname@example.org:group/project.git but the mirroring URL for this is: ssh://email@example.com/group/project.git. The difference lies between the server, git.example.com, and path. When cloning, the separator is a colon, ‘:’. When mirroring, it’s a slash, ‘/’. Getting the authentication can also be tricky. Mirroring more than a few repositories using SSH can become tricky because gitlabs generates a new ssh key for each repository. This is one place of the few places where I like git+https more than git+ssh. Finally, git+https is not without its pitfalls. If like me you also have your own CA then you have the additional problem that git doesn’t do a good job configuring curl’s CA. You have two choices here. On the box initiating the transfer, run:
git config --global http.sslCAPath my-ca-path to use your CA dir or
git config --global http.sslCAInfo my-ca-file.pem to configure your CA file. One advantage of git+https in this configuration is that you can create a single user token for all of your mirroring. My concern with git+ssh here is the proliferation of keys may eventually cause git to fail with a “too many authentication attempts” error. With git+http, you can create one mirroring token for all mirror operations.
Once you’ve setup mirroring, you have a great plan B if the day ever comes that your gitlabs server becomes unavailable. You should have a working gitlabs mirror that you can use in any way that you please. You can even pull backups of the mirror server so you have a redundant, offsite-backup.
It’s been a frustrating week. If it can break, it has broken and lately I’ve been shining up my
ansible to fix it. So I find myself trying to use my shiny new playbooks to address problems and to get my machines to all line up. Today my
ansible-playbook ... run hung up on an arm based mini-nas that I have in my vacation house. My first assumption was that ansible was the problem That was wrong. To find the problem, I ran the playbook and then logged onto the machine seperately. A quick
ps alx gave me this little snippet:
1001 43918 43917 2 52 0 12832 2076 pause Is 1 0:00.03 -ksh (ksh)
0 43943 43918 3 24 0 18200 6916 select I 1 0:00.04 sudo su -
0 43946 43943 2 26 0 13516 2776 wait I 1 0:00.02 su -
0 43947 43946 2 20 0 12832 2024 pause S 1 0:00.03 -su (ksh)
0 51594 43947 3 20 0 13464 2572 - R+ 1 0:00.01 ps alx
0 51578 51527 2 52 0 12832 1980 pause Is+ 0 0:00.01 ksh -c /bin/sh -c '/usr/local/bin/python3.9 /root/.ansible/tmp/ansible-tmp-1694615369.904476-9336-34642038817669/Ansib
0 51579 51578 3 52 0 13536 2552 wait I+ 0 0:00.01 /bin/sh -c /usr/local/bin/python3.9 /root/.ansible/tmp/ansible-tmp-1694615369.904476-9336-34642038817669/AnsiballZ_pkg
0 51580 51579 3 40 0 36756 23668 select I+ 0 0:01.51 /usr/local/bin/python3.9 /root/.ansible/tmp/ansible-tmp-1694615369.904476-9336-34642038817669/AnsiballZ_pkgng.py
0 51582 51580 0 52 0 21388 9048 wait I+ 0 0:00.04 /usr/sbin/pkg update
0 51583 51582 1 52 0 21708 10104 ttyin I+ 0 0:00.19 /usr/sbin/pkg update
This is relevant because because it traces the process tree from my ssh login all the down to the process that’s hung up. Note well that the
pkg update run at PID 51583 is in a ttyin state. Running
pkg update manually gave me this:
# pkg update
Updating FreeBSD repository catalogue...
Fetching packagesite.pkg: 100% 6 MiB 3.3MB/s 00:02
Processing entries: 0%
Newer FreeBSD version for package zziplib:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1302001
- running kernel: 1301000
Ignore the mismatch and continue? [y/N]:
The why of all this doesn’t really matter much. In this case the machine is running a copy of FreeBSD that’s stale, 13.1, and pkgng is asking my permission to update to a package repository from FreeBSD 13.2. What’s important here is a basic debugging technique. The important question is: How does ansible actually work under the covers? The answer is, each ansible builtin prepares a 100k or so blob of python that it spits in …/.ansible/tmp on the remote machine. Then it uses the local python interpreter to run that blob. The python within the blob idempotently does the work. My blob needed to verify that the sudo package on my box. For reasons that I don’t understand but also really don’t mind, it wanted to make sure that the local package collection was up to date. It’s not normal for a box to hang on
pkg update but it’s not crazy either.
I found the
setup.py file for the pyvmomi module here: “https://github.com/vmware/pyvmomi/blob/master/setup.py“. It looks like a reasonable example of what to do in setup.py.
When I was playing with git checkout of modules I discovered that git doesn’t know how to set the certificate store for curl when it tries to retrieves a module via https. In general, I don’t recommend using git with https unless you have to. Using git+ssh obviates away a bucket of authentication issues. In this case, https is the better choice. To tell git where to look for certificates, to verify and https website, I had to add the following to my
The command that does this is:
git config --global http.sslCAPath "/etc/ssl/certs". If your operating system uses a CA file rather than a CA directory this is the setting:
git config --global http.sslCAInfo "/etc/ssl/cert.pem". You can also make this work by setting an environment variable for curl in
Mirroring in Gitlab
I normally strongly prefer
git+https. If you are mirroring between two gitlab-ce instances over git+https, you can handle your mirroring with a single authentication token.
Git Sparse Checkout
At work we had a very large monorepo. I’m tempted to quote Douglas Adams here but the reference is good enough. Checking out the whole thing runs the possibility of confusing git status messages as a result changes in the other part of the tree. These messages are a distraction. Dealing with them can consume large amounts of time. The best way to avoid them perform what’s called a sparse checkout. This is a checkout that only puts what you need into your working directory. In a normal checkout:
$ git clone ...
You get the entire code base in your working directory. A sparse checkout is more complicated to perform:
$ mkdir _target directory_
$ cd _target directory_
$ git init .
$ git config core.sparsecheckout true
$ echo "_your desired subdir_" >> .git/info/sparse-checkout
$ ## Repeat the echo for each directory you need.
$ git remote add origin https://git.neopost.com/PPT/IBMHSM.git
$ git fetch
$ git checkout master
It’s eight steps but if you do it this way, you gain complete control over what’s in your working directory.
I saw a configuration for bash tab completion a few years ago and I’ve always wanted it for the korn shell. I use either the “true” ksh from AT&T via David Korn or one of the variants that has sprung out of the pdksh project. OpenBSD’s ksh is a descendant of pdksh. In a recent release of OpenBSD someone patched it to kludge configurable tab completion via environmental arrays. The article is here: https://www.vincentdelft.be/post/post_20210102
This ksh is shells/oksh in FreeBSD.
I don’t think I’m an Apple fan boy but I definitely like their stuff. That said, everyone could stand some improvement. One place Apple could improve is resets, and general controls.
I used to run a desktop machine 24/7 as an NFS server for backups. During the summer this heats up my office almost to the point where I can’t use it. I decided to change the Desktop to only be on when I need and to be suspended or off when idle. something has to be a target for the backups though. I chose a Mac Mini that I had. I brought the mini to my office and refreshed the machine with a new Operating System. Then I moved the NFS service to it. Finally, I turned off the desktop.
The Mini runs headless. At somepoint, Apple decided to be “helpful”. they changed OS X so that machines without any keyboard at all will plead for a bluetooth keyboard and mouse at periodically. The keyboard that connected to my work laptop obliged and connected and paired without a pin code exchange.
This morning I spent 20 minutes of my time trying to figure out why my keyboard didn’t work before I realized that pairing without a pin code was even possible.
I get that pairing in this manner reduces support calls and I understand that my use case of a Mac Mini running headless is a corner case by far but decisions that theOS X UI team have made lately really don’t help me. I disconnected the keyboard and turned off bluetooth on the mini but I’ll be surprised if that’s the end of things. Apple’s in the midst of a war on wires these days so if you connect your computer using ethernet and turn of wifi, your machine will pester you forever to turn the wifi back on. I’ll see if bluetooth is the same.
Everyone should be using the built in python argparser. This is a good example: https://github.com/vmware/pyvmomi-community-samples/blob/master/samples/list_datastore_info.py