FreeBSD’s geom makes life easier.

FreeBSD’s geom is the missing link that I’ve been searching for. Geom is an abstraction for mass storage providers and consumers which really cleans up the mass storage layer in Unix. In geom your disk drive combined with it’s driver is a mass storage provider. The filesystem layer is a mass storage consumer. Geom provides the glue in between providers and consumers which allows greatly enhanced function. Encrypted or compressed storage can be easily built using this framework.

I’m using geom with the automounter, amd to revamp my use of removable media. If you use the new gnome framework gnome’s hald does this also but it requires the user to unmount the drive. I like using amd here. Even though there’s a slight performance penalty amd will automatically unmount the drive for you after a configured period of inactivity. I find that this is an effective way of getting around one of Unix’s quirks. Unix doesn’t react well with a piece of the filesystem just disappears which is what happens when you remove a mounted USB stick. Amd can be configured to unmount the stick 30 seconds after you’ve stopped using it. Doing this almost eliminated the resulting kernel panics and length fscks that I experienced when I first started using USB storage.

The automounter is a utility that was designed to mount storage into the filesystem on demand. It works by providing an NFS look alike storage system. You literally mount the automounter into a directory in the filesystem. The automounter is configured with a map that assigns different directories within it’s filesystem to different pieces of mounted storage. So if amd is providing the directory /Volumes and has a mapping for MyUsbStick when you try to get a directory listing of /Volumes/MyUsbStick, the automounter has all the information needed to do the mount and provide access to the filesystem beyond. So far that’s about even.

Without geom that was a nice enough setup. But when the kernel attaches your usb stick in FreeBSD it gives it a name like /dev/da0s2e or /dev/da1s1. It’s easy enough to figure out what this means. da0s2e is the “e” partition on the second slice of the first “SCSI” drive in the system. USB and firewire drives a psuedo SCSI drives in FreeBSD and in Linux because the SCSI protocol was flexible enough to serve as the model for mass storage. But what happens if you have two systems and one of them has a SCSI controller in it or if you have two USB sticks. SCSI drives are number in the order in which they appear on the system so for the man with two USB sticks the whether they are connected as Drive a is da0s2e and drive b is da1s2e or vice versa depends on the order in which you plug them in. It turns out that geom embraces the concept of a volume label and can create a shadow device based on that label.

To use this feature of geom load the kernel module geom_vol_ffs (Available in FreeBSD from 6.x onwards) then add a label to your FreeBSD filesystems using the tunefs command:

# tunefs -L “MyLabel” /dev/da0s2e

Then you should end up with a device entry for your disk called /dev/vol/MyLabel

Have fun

— Chris

FreeBSD/WPA

Since I got my Mac some of my FreeBSD projects have been languishing on the back burner. Two are important, getting an IPSEC tunnel using IKE between FreeBSD (racoon) and OpenBSD (isakmpd frontended by ipsecctl) and getting WPA going. A couple of months ago I replaced WEP with wpa in my home wifi setup. There’s no arguing that the security is better and on the Mac it’s drop dead simple. I never understood what was going on in FreeBSD I understand it now. WPA appears to be divided into two parts like IKE. One part runs on the client and another in the Wireless AP. FreeBSD includes a program called wpa_supplicant which manages the WPA key exchange for you. To handle this it also has to manage the wireless interface. The automatic setup is actually pretty easy. I found this which helped me out. I wanted to understand what was going on under the hood. It turns out the setting up the config per the original article is the first step. Then run:

wpa_supplicant -B -Dbsd -iath0

as root. This handles the WPA negotiation. When ifconfig reports that you are connected you can run dhclient ath0 to connect.

Dav backended web servers

My father-in-law asked a question about email the other day that perplexed me. We recently got him a Mac and he was complaining about the “spinning beach ball of death” when he tried to send a message. It turns out that the message was 50 full sized digital photos for my brother-in-law. After he explained that detail it made sense that Mail was choking. I told him that the mail was simply too big and that the better way to share the pictures would be to post them in a website. I also promised to come up with a solution for him in a week or so and give him a helping hand getting everything posted.

Normal people would look at Picasa or Flickr or something but neither of those sites interest me. Most of the photo sites on the web  want you to sign up which infringes on my privacy. They grant you some promotion through search engines for the privilege but Bapa doesn’t need the world to be able to see his photo album. To my “Open source guy” brain the immediate solution would be to create the site and then setup something like coppermine or gallery2 but both of those solutions require me to teach Bapa how to post to a photo web site. Picasa and Flickr have the same issue.

Fortunately for me, Bapa has a Mac. The Mac has a set of wonderfully integrated tools for doing exactly this sort of thing. I decided that the easiest thing to do would be to Apache and mod_dav to grant access to some web space. And then use apache to publish a site from within the web space. The Apache config is actually pretty simple. You use two virtual hosts. One points to the backend and one points the actual site. Here’s the apache2 config for the backend:

<VirtualHost *:80>
    ServerName      backend.example.com
    ServerAlias     backend

    ServerAdmin     webmaster@example.com

    DocumentRoot    /home/www/sites/backend.example.com/
    DAVLockDB       /var/www/DavLockDb

    <Directory "/home/www/sites/backend.example.com/Dav/">
        Dav On
        AuthName    "Photo DAV Fileshare"
        AuthType    Basic
        AuthLDAPURL ldap://ldap-slave.example.com/ou=people,dc=example,dc=com?uid
        <LimitExcept GET HEAD OPTIONS PROPFIND>
            require valid-user
        </LimitExcept>
    </Directory>

    ErrorLog        /home/www/sites/backend.example.com/log/error_log
    CustomLog       /home/www/sites/backend.example.com/log/access_log common
</VirtualHost>

The goal here is to provide DAV access to the web sites storage so the Macintosh toolkit sees it as just another place to store files. The Directory setting tell Apache to provide the Dav directory and everything below it as a shared file system using the DAV protocol. In this case authentication is provided via LDAP but that could easily be changed to .htaccess files.

The actual web sites virtual host config is here.The trick is in setting DocumentRoot to be a subdirectory of the Dav provided above.

<VirtualHost *:80>
    ServerName      photos.example.com
    ServerAlias     photos

    ServerAdmin     bapa@example.com
    DocumentRoot    /home/www/sites/backend.example.com/Dav/photos

    ErrorLog        /home/www/sites/backend.example.com/log/error_log
    CustomLog       /home/www/sites/backend.example.com/log/access_log common
</VirtualHost>

The result is that Bapa can attach the file structure that contains his photo share as a DAV imported file system from his Mac (Finder: Go -> Connect to Serverhttp://backend.example.com/Dav/photos. Then tell iWeb to publish his website into that space et voilá: his pages are magically published to the web under photos.example.com.

This is about 90% of your setup. You will want to secure things to make sure that they aren’t easily broken. Apache accesses the Dav as user: www so it’s pretty important to make sure that the DAV directory is protected from tampering by having it owned by root:

# mkdir -p /home/www/sites/backend.example.com/Dav/photos
# chown root:root /home/www/sites/backend.example.com/Dav
# chmod 775 /home/www/sites/backend.example.com/Dav
# chown root:www /home/www/sites/backend.example.com/Dav/photos
# chmod 775 /home/www/sites/backend.example.com/Dav/photos

These permissions setup the Dav share so that an object stored in the root of the Dav may only be deleted or removed by its owner. The DocumentRoot of the photos site can be written into by the www user. This isn’t meant as absolution security as much as a means to keep down support calls.

Caveats

In this example I’m doing nothing to protect Bapa’s password from interlopers on the net. In reality you would want the DAV to be provided over https to protect the password. A VPN connection is another possibility.

Old Code does die

I have a very old VA Linux 2200 box that I use a firewall. I recently upgraded it a later version of OpenBSD but it appears that I’ve found a regression in the X Server. This machine uses the Intel 440Gx Chipset with an integrated Cirrus Logic CL GD5480 Video adapter. It looks like the support for the video adapter has fallen out of Xorg 7.2 since the old OpenBSD could drive this box at 1280x1024x16bpp even though the box only has 2M of video RAM (If you do the math, don’t ask me I’m trying to found out how myself).  The new driver can’t do this. I’ve spent a few hours trying to find Doco for the chipset in Xorg but the man page is another one of those “This section needs to be completed things…”

Use the source, Luke!

I wrote earlier about SASL and postfix. One side affect of my setup has been that I get these spurious warnings in my logs.

Apr 4 10:22:19 corellia postfix/smtpd[69626]: auxpropfunc error invalid parameter supplied

I’ve been meaning to throw some time at this problem for a while now but everything works so I haven’t. Upgrading my infrastructure to the latest Open and FreeBSD’s has me using newer packet filtering code with more capabilities so what was once a non-problem has become a pain in the neck. This problem is tied to another feature of cyrus sasl that if find annoying. The configuration for postfix and cyrus is handled through a file called smtpd.conf. This file is stored in /usr/local/lib/sasl2/smtpd.conf. This is annoying for one because under Unix configuration files like this belong in /etc. But for two because the file is poorly documented at best. Reading the source for postfix shows that this is handled by the smtpd_sasl_path. It’s already well documented that this variable isn’t a path, it’s the base file name for the configuration file. This is fixed in postfix 2.5. The warning comes from the initialization of the ldapdb component of sasl. Even though I’m not using it I have to specify the parameter ldapdb_uri.

Keeping up to date with FreeBSD

I use FreeBSD for nearly anything that needs a server. It’s got quite a bit to offer. Anyone who actually knows FreeBSD knows that it’s dead simple keep up to date. I’ve used this basic technique for several years.  The steps are pretty simple and can be found here with additional instructions for dealing with multiple machines here.  I’ve pretty much followed this method for years including updating a machine located in a remote close with an NFS mounted /usr/{src,obj} over an IPSEC link. I recently added a new wrinkle that I think is pretty cool. My build box is an HP/Compaq DL360 with hardware RAID. I’ve pretty much standardized on this hardware. A client clued me into this simple technique. Long story short he had to move a data center from Chicago to CT and he chose to do it by stocking up on spare RAID drives. He cloned a server by pulling a working drive from a working server and replacing it with a spare. He shipped the pulled drive via courier to the new data center. Installed it in the correct slot on the same kind of server chassis booted the new clone up. At this point the clone server now saw it’s drive array missing the other drive. He was obviously mirroring. On insertion of the new drive the clone server hardware did an automatic rebuild. I apply the same technique to FreeBSD. I build a new server from the lastest snapshot then go through the source update process. Next I pull one of the drives and put the pull aside replacing it with a spare.  Et voila. Now I have a save point on that Drive. I can install it in the server alone an reboot. and I’m right at the point where I have built and installed the world and have just finished running mergemaster. This is an excellent starting point for building a fresh server.

Lil’ Bobby Tables

A while back I found xkcd. I found it through a mailing list but this post grabbed my attention. Here’s a recent comic which made me think about the way that popular web applications use SQL databases. If you don’t get it the
Mom in the comic executed a classic SQL injection attack against her son’s school. (BTW I didn’t want to spoil the joke. You can see the previous sentence by highlighting it with your mouse.) In any case this is a common attack method against many current web applications and it shows just how naively many of the programmers are of SQL in general. There are two practical ways to defend against this attack. The most obvious is to validate your SQL before you pass it to the database engine. One that requires a little more thought would be to disallow the web database user from being able to DROP TABLES in the first place. Any real web application should expect a database with at least two users, root, or dba and webuser, or www. Root should be allowed to do anything to the database but his credentials need to be protected. If your web application has grown to the point where you’ve split your database server from your webserver for performance purposes. Allowing the root or dba level of access from localhost only is a good start. Webuser should be able to SELECT on your applications tables. He should be able to INSERT, UPDATE, and DELETE on a limited subset of tables as your database will allow. He may need to be able to CREATE a temporary table and possible DROP the same but that’s a job that’s really better done by a stored procedure. E.g. create a stored procedure that does the needed manipulation and then allow the webuser the privilege to call the stored. Obviously what you can do here depends on your database. I know that Postgresql can grant these very fine grained security settings. If I recall correctly MySQL is a little more course but is still workable.

Hylafax: Ugh

I tried to setup hylafax today. I had it going a few years ago. I even had a neat hack where I would have it take all inbound faxes, convert them into pdf and store them in directory accessible from the web. It was pretty cool. I figured I’d re-create that and maybe add some Python-Fu to have an outbound directory but alas it wasn’t meant to be. I ran around in circles for three hours trying to eliminate the problems but got no-where until I installed efax and right off the bat the fax just worked. That eliminated the problems of (The modem broke between last time and now, the modem doesn’t like the VoIP line, and my new HP A-I-O doesn’t like the fax modem) leaving Hylafax is misconfigured. Here we go again, another mailing list……

More stuff about Postgresql that should be obvious.

I’ve been scratching my head on this one for far too long. I have a query under Postgresql which retrieves the distance between too points given the knowledge of their zipcodes. This work because I have an incomplete table of mileages between arbitrary three digit zipcode pairs. Each time I use this table my queries take a long time and I could never understand why. It has to do with the type that postgresql assigns to computed text fields. I was doing something like this:

SELECT * FROM worklist INNER JOIN partial_zipcode_mileage ON SUBSTRING(worklist.origin_zipcode, 1, 3) = partial_zipcode_mileage.origin_partial_zipcode...

The issue here is the type of the expression: SUBSTRING(worklist.origin_zipcode, 1, 3) as compared to the type of the field partial_zipcode_mileage.origin_partial_zipcode. The latter is a SQL CHAR(3) since it will always hold 3 characters. Postgresql assignes the first expression a type of TEXT since it has know way to know how bit a field you actually want. This prevents the postgresql engine from using the index and this, my query takes along time. Substitute SUBSTRING(worklist.origin_zipcode, 1, 3)::char(3) in the statement and all is happy.

Exception trapping in pl/pgsql.

Learned a new trick with postgresql stored procedures today. This will probably appear to be obvious but it’s new to me. You can do exception trapping in pl/pgsql by you can also ignore some errors. The form is:


BEGIN
DROP TABLE foo;
EXCEPTION
WHEN undefined_table THEN NULL;
WHEN OTHERS THEN
RAISE NOTICE, ('Notice: ' || SQLSTATE || ' - ' || SQLERRM);
END;