Looks like something burped on my mailserver and my bogofilter wordlist got too big. Probably something to do with limits anyhow. In any case I was looking for a way to recover from the issue and came across this pearl in the Bogofilter FAQ. Well, the advice is incomplete. If you really hose up the database then bogoutil -d will stop printing entries before the end of the database. The next recovery step is to use the db utilities: db_dump and db_load to fix the database. db_dump -r (on FreeBSD db_dump-<version>) dumps the database into a text file and db_load creates a text file from a word list. The problem is that the advice in the bogofilter faq is out of date. It looks like there are some parameters that have to be specified. My solution: use db_dump without the -r that creates a broken database with a default header. Copy the header into the new text file and then append the output of db_dump -r to that. Et voila!
Mother of all MiFi wishlist
My Mother of all MiFi wishlist:
- Runs for 4 ~ 5 hours on rechargable batteries. Preferably 4xAA NiMh cells which I have in abundance.
- WPA encryption if possible otherwise pre-auth by mac address or live auth via authpf.
- Automatically connects to my lan using certificate based IPSec.
- Provides DNS locally.
- Gui configuration but can be a python TkInter of X11 Gui.
- 802.11b/g although given my experience last week 802.11n over 5GHz would be nice.
- SNMP configuration? That’s why I got an enterprise number from IETF.
To Do:
- Put the Soekris Net4511 on my Kill-a-watt meter to see how much juice it really needs (and how efficient the power supply is.)
- Figure out how to get USB into the thing. The outside internet will be a Verizon or Sprint network dongle.
- Get a case and power supply for the 4511
- Will OpenBSD provide WPA2 authentication?
- How hard is it going to be to get a USB jack into a 4511 case? (Bill Johnson?)
- How many people can I connect to it before it’s overloaded?
- 4521 Case? Automatically has room for batteries.
Alright, it’s no longer 1998!
One thing that really ticks me off the web designer conversation where your web design guy insists on designing to an 800×600 screen resolution to ensure that your pages will be accessible by everyone on the web. Today I ran across this nugget (opens in a new window). I’ve always said that this is so 1998 yet I’ve had this conversation as recently as 2007. Well, if you dissect the table you come up with this:
Width | |
1920×1200 | 2.27% |
1680×1050 | 8.72% |
1440×900 | 18.37% |
1366×768 | 20.76% |
1280×1024 | —- |
1280×800 | —- |
1280×768 | 58.09% |
1152×864 | 61.04% |
1024×768 | 94.94% |
800×600 | 100.00% |
Height | |
1920×1200 | 2.27% |
1680×1050 | 8.72% |
1280×1024 | 21.97% |
1440×900 | 31.62% |
1152×864 | 34.57% |
1280×800 | 56.92% |
1366×768 | —- |
1280×768 | —- |
1024×768 | 94.94% |
800×600 | 100.00% |
That’s right. If you design for 1024×768 you reaching nearly 95% of all the web browsers that participated in this survey. Now web designers can partly like it’s 2004!
How to torpedo your own open source project.
My friend Leanne has a wisdom that I am only beginning to understand. A couple of weeks ago we were having pints of beer at an old college hangout and topic of conversation got onto computers. At this point a mutual friend interjected:
“But the Macintosh is the best computer platform available. How can you not think it will take over the entire PC marketplace?”
— DW
Now, if someone accuses me of being a Mac fanboi I’ll demure so I agree with his statement about the best platform. But I’m more than willing to argue over his question. As I learned long ago from Leanne:
 “Being the best product in the market place does not guarantee that you will win in your market.”
–LF
 As denizens of open source this is an axiom that we would do well to understand. Most time when I meet Linux fanbois, especially in businesses outside of the computer industry I both admire and am appalled by some of the things that they want to do. I wish they would ask themselves the question:
 If Microsoft is not the best product in it’s marketplace why does it have the best marketshare?Â
In operating systems Microsoft is certainly not the best product in the marketplace. It has the most users because of a combination of things. Not to be minimized in that list of attributes is the support that Microsoft gives to it’s customers. (In fact support is probably the main reason why Mac OS X is the best product in the OS market).
 For denizens of Open Source the support line is critical. As this economy rolls onward more and more of us will be getting the opportunity to build cool business projects on Open Source. When we do this we must understand that being the best tool in the market place isn’t enough. So when you are making the decision between say FreeBSD, CentOS, and Red Hat Linux: Unless you know that you or someone a lot like you will be supporting the project until you are 90 years old and buried. You’d better choose Red Hat or have a damned good reason for choosing something else. As a consultant I’m getting used to seeing and hearing IT managers freak out when they find out that some mission critical app is running on an operating system that they haven’t heard of. Trust me, you are very lucky if your Manager is even aware of CentOS.
 If you must choose an open source application for your project it’s very important that the application has developed to the point of having both a stable and a development branch. Because no matter how sexy the development branch is, if you want to see the project end up with a reasonable life span then you’d better choose the stable one.
 These are two of the things that your Director of Open Source Development and CIO want to see. They want to see it for a lot of reasons:
 Imagine, a mission critical app is intermittently up and down because of a bug in the operating system. Two managers, one’s on the phone to Red Hat in North Carolina and the other is grovelling through Google looking for a source code kernel patch for FreeBSD. Which one gets to keep his job after the problem is fixed?
 Imagine, your intranet project that needs just a couple of tweaks to authenticate against Active Directory so it can go corporate. The expertise to do so doesn’t exist in house because you left for that cherry job at Google. What does your manager say when he finds out from his consultant that you need another 20 man hours to pull the project onto a stable release the wiki software that you chose for the foundation?
 Peace
Dante
In Dante’s Inferno there were circles in hell designed to separate the ordinary sinner: the guy who designed the keyboard I’m working with (which provides no feedback when a key has been struck for example) from the guy who deliberately put the “global nuclear war” button right next to the “toast apple poptarts” button. My “9th circle of hell award” goes to the guys who designed the firewall that I’m working with lately. It appears that in their wisdom they’ve chosen to implement the “Red Alert — all hands on deck” alarm for the following scenario. You have a server connected to a client via tcp. The server is a fairly recent linux box that can do RFC1323 extensions. The client is a boring Windows XP box with a TCP RWin size of 65536 bytes. Between them is a Comcast business class Cable connection. In this scenerio the Windows box is trying to download a file from the server on the Comcast connection. The problem is literally that the connection is too fast for the Windows XP Box to fully cope. Nowadays when I test Comcast Cable connections I’m surprised to see anything less than 25Mbit/s.In whole numbers thats 25,000,o000 bits / sec. In more familiar units that 312.5 kBytes /s. The problem is that I’m starting to see firewalls that see this as an issue because they have been programmed with very conservative specifications about what constitutes a denial of service attack. I’m seeing firewalls that scream DOS when they are connected to a Business Cable modem line and have clients with tcp receive window size of 65536 bytes. Why? it’s simple. On aBusiness Cable line with 25Mbits/s download rate you have to be able to buffer 96kbytes/s in tcp windows just to keep up with a server (or client) at the other end of a fast line. These firewalls are calling DOS because the other end can fill their TCP window and then some. The right thing to do is to watch. If the otherside wants to DOS you he’ll send many packets after your Rwin is filled. If he’s just a really fast server on a really fast pipe. He’ll respect your RWin and quit sending. If you’re firewall decides to be agressive and drop the connection (by proactively sending a TCP RST) then you should probably act accordingly.
My thanks to Chuck Skuba on this post. I have to be 100% and fess up that I gathered the data but he did the homework.
— Chris
FreeBSD’s geom makes life easier.
FreeBSD’s geom is the missing link that I’ve been searching for. Geom is an abstraction for mass storage providers and consumers which really cleans up the mass storage layer in Unix. In geom your disk drive combined with it’s driver is a mass storage provider. The filesystem layer is a mass storage consumer. Geom provides the glue in between providers and consumers which allows greatly enhanced function. Encrypted or compressed storage can be easily built using this framework.
I’m using geom with the automounter, amd to revamp my use of removable media. If you use the new gnome framework gnome’s hald does this also but it requires the user to unmount the drive. I like using amd here. Even though there’s a slight performance penalty amd will automatically unmount the drive for you after a configured period of inactivity. I find that this is an effective way of getting around one of Unix’s quirks. Unix doesn’t react well with a piece of the filesystem just disappears which is what happens when you remove a mounted USB stick. Amd can be configured to unmount the stick 30 seconds after you’ve stopped using it. Doing this almost eliminated the resulting kernel panics and length fscks that I experienced when I first started using USB storage.
The automounter is a utility that was designed to mount storage into the filesystem on demand. It works by providing an NFS look alike storage system. You literally mount the automounter into a directory in the filesystem. The automounter is configured with a map that assigns different directories within it’s filesystem to different pieces of mounted storage. So if amd is providing the directory /Volumes and has a mapping for MyUsbStick when you try to get a directory listing of /Volumes/MyUsbStick, the automounter has all the information needed to do the mount and provide access to the filesystem beyond. So far that’s about even.
Without geom that was a nice enough setup. But when the kernel attaches your usb stick in FreeBSD it gives it a name like /dev/da0s2e or /dev/da1s1. It’s easy enough to figure out what this means. da0s2e is the “e” partition on the second slice of the first “SCSI” drive in the system. USB and firewire drives a psuedo SCSI drives in FreeBSD and in Linux because the SCSI protocol was flexible enough to serve as the model for mass storage. But what happens if you have two systems and one of them has a SCSI controller in it or if you have two USB sticks. SCSI drives are number in the order in which they appear on the system so for the man with two USB sticks the whether they are connected as Drive a is da0s2e and drive b is da1s2e or vice versa depends on the order in which you plug them in. It turns out that geom embraces the concept of a volume label and can create a shadow device based on that label.
To use this feature of geom load the kernel module geom_vol_ffs (Available in FreeBSD from 6.x onwards) then add a label to your FreeBSD filesystems using the tunefs command:
# tunefs -L “MyLabel” /dev/da0s2e
Then you should end up with a device entry for your disk called /dev/vol/MyLabel
Have fun
— Chris
FreeBSD/WPA
Since I got my Mac some of my FreeBSD projects have been languishing on the back burner. Two are important, getting an IPSEC tunnel using IKE between FreeBSD (racoon) and OpenBSD (isakmpd frontended by ipsecctl) and getting WPA going. A couple of months ago I replaced WEP with wpa in my home wifi setup. There’s no arguing that the security is better and on the Mac it’s drop dead simple. I never understood what was going on in FreeBSD I understand it now. WPA appears to be divided into two parts like IKE. One part runs on the client and another in the Wireless AP. FreeBSD includes a program called wpa_supplicant which manages the WPA key exchange for you. To handle this it also has to manage the wireless interface. The automatic setup is actually pretty easy. I found this which helped me out. I wanted to understand what was going on under the hood. It turns out the setting up the config per the original article is the first step. Then run:
wpa_supplicant -B -Dbsd -iath0
as root. This handles the WPA negotiation. When ifconfig reports that you are connected you can run dhclient ath0 to connect.
Dav backended web servers
My father-in-law asked a question about email the other day that perplexed me. We recently got him a Mac and he was complaining about the “spinning beach ball of death” when he tried to send a message. It turns out that the message was 50 full sized digital photos for my brother-in-law. After he explained that detail it made sense that Mail was choking. I told him that the mail was simply too big and that the better way to share the pictures would be to post them in a website. I also promised to come up with a solution for him in a week or so and give him a helping hand getting everything posted.
Normal people would look at Picasa or Flickr or something but neither of those sites interest me. Most of the photo sites on the web want you to sign up which infringes on my privacy. They grant you some promotion through search engines for the privilege but Bapa doesn’t need the world to be able to see his photo album. To my “Open source guy” brain the immediate solution would be to create the site and then setup something like coppermine or gallery2 but both of those solutions require me to teach Bapa how to post to a photo web site. Picasa and Flickr have the same issue.
Fortunately for me, Bapa has a Mac. The Mac has a set of wonderfully integrated tools for doing exactly this sort of thing. I decided that the easiest thing to do would be to Apache and mod_dav to grant access to some web space. And then use apache to publish a site from within the web space. The Apache config is actually pretty simple. You use two virtual hosts. One points to the backend and one points the actual site. Here’s the apache2 config for the backend:
<VirtualHost *:80> ServerName backend.example.com ServerAlias backend ServerAdmin webmaster@example.com DocumentRoot /home/www/sites/backend.example.com/ DAVLockDB /var/www/DavLockDb <Directory "/home/www/sites/backend.example.com/Dav/"> Dav On AuthName "Photo DAV Fileshare" AuthType Basic AuthLDAPURL ldap://ldap-slave.example.com/ou=people,dc=example,dc=com?uid <LimitExcept GET HEAD OPTIONS PROPFIND> require valid-user </LimitExcept> </Directory> ErrorLog /home/www/sites/backend.example.com/log/error_log CustomLog /home/www/sites/backend.example.com/log/access_log common </VirtualHost>
The goal here is to provide DAV access to the web sites storage so the Macintosh toolkit sees it as just another place to store files. The Directory setting tell Apache to provide the Dav directory and everything below it as a shared file system using the DAV protocol. In this case authentication is provided via LDAP but that could easily be changed to .htaccess files.
The actual web sites virtual host config is here.The trick is in setting DocumentRoot to be a subdirectory of the Dav provided above.
<VirtualHost *:80> ServerName photos.example.com ServerAlias photos ServerAdmin bapa@example.com DocumentRoot /home/www/sites/backend.example.com/Dav/photos ErrorLog /home/www/sites/backend.example.com/log/error_log CustomLog /home/www/sites/backend.example.com/log/access_log common </VirtualHost>
The result is that Bapa can attach the file structure that contains his photo share as a DAV imported file system from his Mac (Finder: Go -> Connect to Server – http://backend.example.com/Dav/photos. Then tell iWeb to publish his website into that space et voilá: his pages are magically published to the web under photos.example.com.
This is about 90% of your setup. You will want to secure things to make sure that they aren’t easily broken. Apache accesses the Dav as user: www so it’s pretty important to make sure that the DAV directory is protected from tampering by having it owned by root:
# mkdir -p /home/www/sites/backend.example.com/Dav/photos # chown root:root /home/www/sites/backend.example.com/Dav # chmod 775 /home/www/sites/backend.example.com/Dav
# chown root:www /home/www/sites/backend.example.com/Dav/photos # chmod 775 /home/www/sites/backend.example.com/Dav/photos
These permissions setup the Dav share so that an object stored in the root of the Dav may only be deleted or removed by its owner. The DocumentRoot of the photos site can be written into by the www user. This isn’t meant as absolution security as much as a means to keep down support calls.
Caveats
In this example I’m doing nothing to protect Bapa’s password from interlopers on the net. In reality you would want the DAV to be provided over https to protect the password. A VPN connection is another possibility.
Old Code does die
I have a very old VA Linux 2200 box that I use a firewall. I recently upgraded it a later version of OpenBSD but it appears that I’ve found a regression in the X Server. This machine uses the Intel 440Gx Chipset with an integrated Cirrus Logic CL GD5480 Video adapter. It looks like the support for the video adapter has fallen out of Xorg 7.2 since the old OpenBSD could drive this box at 1280x1024x16bpp even though the box only has 2M of video RAM (If you do the math, don’t ask me I’m trying to found out how myself). The new driver can’t do this. I’ve spent a few hours trying to find Doco for the chipset in Xorg but the man page is another one of those “This section needs to be completed things…”
Use the source, Luke!
I wrote earlier about SASL and postfix. One side affect of my setup has been that I get these spurious warnings in my logs.
Apr 4 10:22:19 corellia postfix/smtpd[69626]: auxpropfunc error invalid parameter supplied
I’ve been meaning to throw some time at this problem for a while now but everything works so I haven’t. Upgrading my infrastructure to the latest Open and FreeBSD’s has me using newer packet filtering code with more capabilities so what was once a non-problem has become a pain in the neck. This problem is tied to another feature of cyrus sasl that if find annoying. The configuration for postfix and cyrus is handled through a file called smtpd.conf. This file is stored in /usr/local/lib/sasl2/smtpd.conf. This is annoying for one because under Unix configuration files like this belong in /etc. But for two because the file is poorly documented at best. Reading the source for postfix shows that this is handled by the smtpd_sasl_path. It’s already well documented that this variable isn’t a path, it’s the base file name for the configuration file. This is fixed in postfix 2.5. The warning comes from the initialization of the ldapdb component of sasl. Even though I’m not using it I have to specify the parameter ldapdb_uri.