Category: Unix

Stuff I’ve learned about Unix.

Posted on

Email Deliver-ability

Way back in the day in 1996, I remember attending a Birds of a feather session at the USENIX technical conference, on email and spam. The people in the room railed at the spam problem and it was clear that the leaders were taking the spam as a personal attack. I sat quietly in the room, silently noting to myself that none of the proposed solutions, not even adding extensions to the SMTP protocol, were going to stop the growing commercialization of email as a medium. This is because any magic dust that you can sprinkle on email to mark is as trustworthy and not spam, can be and will be ruthlessly adopted by commercial senders to increase their own deliver-ability.

Increasing deliver-ability

I just added DKIM signing to messages that come from vindaloo.com. I did this because I added a new domain to my mail server so I could support my wife’s LLC: moderncrc.com. Honestly, I might have been better off outsourcing this to Purely Mail and if you are here trying to figure out how to set up mail for your own domain, I say that for 90% of people, outsourcing to someone like Purely Mail is the right way to go.

For self hosters and smaller companies, considering hosting there own email, consider the fact that deliver-ability will be your biggest problem. This means that getting other people to accept mail from you and not automatically treat it as spam to be quarantined, rather than read, is the biggest hurdle you will have to get over. In the modern internet, achieving deliver-ability means jumping through a few hoops.

  • You need to get an IPv4 address that hasn’t been fouled by someone using it to send spam. When these addresses get fouled, they get enumerated onto lists called RBLs or real-time blackhole lists. These are DNS based lists that say, this IP address could be, a source of spam. This isn’t generally difficult but it means that you won’t ever be able to send SMTP mail from an end-user internet connection such as an Xfinity or FiOS internet account. And being clear, I mean across cable, fiber, and business, or residential. The best way past this hurdle is to setup your outgoing SMTP server on a VPS from someone like vultr.com. After this you’ll probably need to put in a support request to be allowed to send mail at all. Of course, this pretty much means that you need to know how to run a Linux server with all that that entails.
  • You’ll need to setup DNS for your domain at leas SPF, and DMARC, but probably also DKIM. Microsoft, Google, and Yahoo are all requiring DMARC and either SPF or DKIM to deliver your messages. SPF is simple. You just enumerate the IP addresses that you allow to send email from your domain. DKIM is a little harder. You setup a private-key, public-key pair; then for each message that you send, you extract a portion of it and you arrange for your email server to create a signature of the portion using your private key. You publish the public key in your DNS. People receiving your email from you can verify this signature and if it all works, they know that you are the actual sender of the email rather than a spammer.

Where we ended up

All of this generally works but my frustration stems from the fact that it does very little to reduce spam. For years, over 80% of the spam that I receive has had valid SPF and DKIM and I’m writing this today because yet another obvious phishing attempt was send to me. Of course, it passed SPF and DKIM with flying colors.

Thus we end up in a world of unintended consequences. Rather then the internet as envisioned, a large group of equally participating networks, we are slowly moving to a world where only Microsoft, Google, and Yahoo can deliver email.

Posted on

Add the watchdog timer on Raspberry pi

I learned about the kernel watchdog timer when I ran my network appliances on hardware from Soekris Engineering. The appliances should be running unless one has specifically turned them off but as the that’s not the reality of the world. A watchdog timer is something that has to be frobbed, I’ve also seen petted periodically or the kernel will assume that all of userland is borked. When userland is borked this way, the kernel will reset, hoping that a reset will kickstart enough of the userland infrastructure programs to put the system into a workable state. On Soekris, you could program the kernel watchdog to timeout of 60s. On the Pi, it looks like the maximum is 15s. Raspberry Pi’s are weird. For small jobs, the best way to run them is on SD cards.But an SD card in a Pi will die from write exhaustion after running for something between a year and two years. This is my experience based on using SanDisk 8G SD cards without any consideration for write exhaustion. You can mitigate this and lengthen the lifetime in many ways:

  • Purchase a 16G card where you have an 8G need.
  • Purchase a Pi with more RAM than you need and perform write heavy task to a memory backed filesystem or ramdisk.
  • Do all of your logging via syslog and push all your logs onto a remote log server.

On the Pi devices that I use as cameras, I write the picture / movie output to a RAM disk. I have also noticed that the older SD devices I was buying in the past were more susceptible to this problem that the devices that I buy lately. This is probably a biased observation.

Posted on

MacOS disk repairs

If anybody ever says that Apple is a lot better than Microsoft one thing that they need to pay attention to is the fact that both companies are guilty of the same problems. In this case I’m talking about both companies habit of letting long standing regressions in their software languish, unaddressed for long periods of time. Apple’s sin in this case is with Disk Utility. Apples has allowed a bug in the Disk Utility in macOS go on, unaddresses since about OS X 10.13 or so when they changed the structure of Time Machine Backups to force an encrypted drive. I’ll admit that I’m not being completely fair. I’m running an older version of macOS on my laptop so this bug may indeed be fixed but it still stayed in the software for a good 3 years.

By trying to make it easier to use an encrypted volume for backups Apple has added a few steps to the process of checking these volumes for structural errors. This means the graphic Disk Utility frequently false positives, saying that your volume has a problem. The real issue is that Disk Utility hasn’t properly set things up for the volume check to happen. Back in the olden days, UNIX you wouldn’t let you use a Volume with structural problems because you couldn’t mount it with write allowed. Today is you can mount broken drives in write mode. Then you get to cross your fingers that you’re not compounding an existing problem. Side note: Here’s where I admit to being really really old because 99% of the time its actually okay and that’s actually the case. The result is that Disk Utility can’t properly check out your Time Machine Volumes. To check one out you need to take the time to boot your machine into recovery mode where all of this shiny that makes users happy is disabled. In recovery mode, Disk Utility just works. Compounding this problem, when Apple does the check from a normal boot, it doesn’t detect its own bug and declares that your volume is dangerously corrupted and unreliable so your best best is to start from scratch. This article shows how you can at least get some peace of mind by checking the state of the volume and repairing it from the command line in a terminal window. I would’ve liked to have seen a screenshot of the command line session. But the author decided that figuring out which disk you need to check is too difficult and they didn’t include one. That’s the responsible choice since you are going to be doing a lot of potentially destructive commands with sudo. I worked my way through the process on my own third Time Machine Volume. I have this issue because this Volume is connected to my docking station. It auto mounts when I use my machine on my desktop so I can have a full sized monitor. It’s easy to forget that the Volume needs to be ejected cleanly and quiesced before I disconnect from the docking station. I’m cultivating the habit of ejecting this Volume when my backup has completed.

Posted on

OpenBSD’s ksh adds configurable tab completion

I saw a configuration for bash tab completion a few years ago and I’ve always wanted it for the korn shell. I use either the “true” ksh from AT&T via David Korn or one of the variants that has sprung out of the pdksh project. OpenBSD’s ksh is a descendant of pdksh. In a recent release of OpenBSD someone patched it to kludge configurable tab completion via environmental arrays. The article is here: https://www.vincentdelft.be/post/post_20210102

This ksh is shells/oksh in FreeBSD.

Posted on

Setting up OS X Mail

Old hand Unix guys like myself are used to plain text email. OS X mail will gladly oblige me if I go to: Preferences –> Composing –> Message Format: { Plain Text }. But Apple chooses to use a proportionally spaced font: Helvetica by default. To fix this, go to: Preferences –> Fonts & Colors –> Use fixed width font for plain text messages. I think that these two should go hand in hand but Apple thinks otherwise.

Sadly, what you cannot fix is the line width. It would be nice to have the plain text message lines wrapped at the nth column but, I’ll take what I can get.

Posted on

Mutt account passwords

First, to give credit where it’s due, I started here. That said, here’s how I store and access account passwords in mutt on Linux.

## -- Passwords: encrypted by gpg --------------------------------------------------------------

source “/bin/gpg -d ~/.keychain/mutt.password.neopost.gpg 2>/dev/null |”

The source line in gpg tells mutt to decrypt a file at startup. The file .keychain/mutt… contains two mutt configuration lines:

set imap_pass = "<my_email_password>"
set smtp_pass = "<my_email_password>"

I created it as follows:

$ cat <<EOF | gpg -r <my_gpg_id> ~/.keychain/mutt.password.neopost.gpg
set imap_pass = "<my_email_password>"
set smtp_pass = "<my_email_password>"
EOF
$

Gpg knows how to decrypt this file and retrieve the plain text configuration. Note well that I used a “Here” document to create the file. This keeps mail password out of the filesystem. Simple stuff, at mutt startup the first time I use it, gpg-agent asks for my gpg key and unlocks the configuration snippet.

Posted on

Emacs use tabs rather than spaces.

Today, about the only place you should see an ascii TAB in a file is in a Makefile. In a world where memory is metered as gigabytes of RAM and terabytes of storage on fast SSDs there is absolutely no need to save space in a source code or configuration file by using a tab rather than two or four or eight spaces. Note well that I may be talking to your editor configuration and not you. But when you write code you should say what you mean and mean what you say unequivocally. I say this because I have been looking at a whitespace difference in my puppet checks for better than a month now. This is because my file has in production has tabs and my file in the puppet/git repository has spaces.

To that end, I’m linking this bit of Emacs magic for readers and my future self.

Posted on

Submission brutes

Brush aside vandals attacking my submission daemon with a little sed:


submission_brutes=$(bzcat /var/log/maillog.0.bz2 | \
cat - /var/log/maillog | \
sed -Ene '/postfix\/submission\/smtpd.*errors after AUTH/s/^.*[^0-9]+(([0-9]+\.){3}[0-9]*).*$/\1/p' | sort -u)
[[ ! -z "${submission_brutes}" ]] && pfctl -t blackhole -T add ${submission_brutes}