It’s been a frustrating week. If it can break, it has broken and lately I’ve been shining up my ansible to fix it. So I find myself trying to use my shiny new playbooks to address problems and to get my machines to all line up. Today my ansible-playbook ... run hung up on an arm based mini-nas that I have in my vacation house. My first assumption was that ansible was the problem That was wrong. To find the problem, I ran the playbook and then logged onto the machine seperately. A quick ps alx gave me this little snippet:
1001 43918 43917 2 52 0 12832 2076 pause Is 1 0:00.03 -ksh (ksh)
0 43943 43918 3 24 0 18200 6916 select I 1 0:00.04 sudo su -
0 43946 43943 2 26 0 13516 2776 wait I 1 0:00.02 su -
0 43947 43946 2 20 0 12832 2024 pause S 1 0:00.03 -su (ksh)
0 51594 43947 3 20 0 13464 2572 - R+ 1 0:00.01 ps alx
0 51578 51527 2 52 0 12832 1980 pause Is+ 0 0:00.01 ksh -c /bin/sh -c '/usr/local/bin/python3.9 /root/.ansible/tmp/ansible-tmp-1694615369.904476-9336-34642038817669/Ansib
0 51579 51578 3 52 0 13536 2552 wait I+ 0 0:00.01 /bin/sh -c /usr/local/bin/python3.9 /root/.ansible/tmp/ansible-tmp-1694615369.904476-9336-34642038817669/AnsiballZ_pkg
0 51580 51579 3 40 0 36756 23668 select I+ 0 0:01.51 /usr/local/bin/python3.9 /root/.ansible/tmp/ansible-tmp-1694615369.904476-9336-34642038817669/AnsiballZ_pkgng.py
0 51582 51580 0 52 0 21388 9048 wait I+ 0 0:00.04 /usr/sbin/pkg update
0 51583 51582 1 52 0 21708 10104 ttyin I+ 0 0:00.19 /usr/sbin/pkg updateThis is relevant because because it traces the process tree from my ssh login all the down to the process that’s hung up. Note well that the pkg update run at PID 51583 is in a ttyin state. Running pkg update manually gave me this:
# pkg update
Updating FreeBSD repository catalogue...
Fetching packagesite.pkg: 100% 6 MiB 3.3MB/s 00:02
Processing entries: 0%
Newer FreeBSD version for package zziplib:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1302001
- running kernel: 1301000
Ignore the mismatch and continue? [y/N]: The why of all this doesn’t really matter much. In this case the machine is running a copy of FreeBSD that’s stale, 13.1, and pkgng is asking my permission to update to a package repository from FreeBSD 13.2. What’s important here is a basic debugging technique. The important question is: How does ansible actually work under the covers? The answer is, each ansible builtin prepares a 100k or so blob of python that it spits in …/.ansible/tmp on the remote machine. Then it uses the local python interpreter to run that blob. The python within the blob idempotently does the work. My blob needed to verify that the sudo package on my box. For reasons that I don’t understand but also really don’t mind, it wanted to make sure that the local package collection was up to date. It’s not normal for a box to hang on pkg update but it’s not crazy either.