Automating the dart league (Part 1)

I’m trying to automate the process of getting scores on my Dart League web page. To that end I’ve reworked the typical dart league standing Spreadsheet (called a model for the rest of the the article) for easier integration with a website.

Here’s the sheet: Dart League Model.

The idea is that the League secretary or statistician will update this model to generate the periodic league standings sheet. The added bonus is that this model has a sheet (see why I wanted to call it a “model”) that serializes the results data for easily and automatically updating the mod_python/postgresql or php/mysql or whatever you prefer. In the long haul I’ll use the following toolset:

  1. php or mod_python to handle the file upload form
  2. mod_python or php to handshake in the uploaded file
  3. xls2csv to convert from Microsoft Excel format to comma separated value format
  4. mod_python to insert the data into a temporary table in mysql or postgresql
  5. mysql or postgresql to integrate the results into the website.

Alternatively I may use procmail  and python to handshake in the data from an email attachment. This replaces steps 1 and 2 above. The result will be either a webform where you upload your statistics or an email address for a bot which automatically handshakes in your stats. If I go email I may have to wrap everything with GnuPG to sign the data so I can know what to ignore.

Just for the purposes of naming I’m going to refer to a “spreadsheet” as a “model” from here on in. This is because Excel allows you to have multiple “sheets” within a model and you can link the data between the sheets. The model that I’m presenting makes pretty heavy use of this feature. The following describes how it works and I’m gonna give you estimates of how long it took me to set things up. I consider myself pretty good at Excel so double any times that I quote for Novice Excel users. If someone here is good at Excel Macros I can see spending some time to embed some of my knowledge into the model in the form of macros.

To use this model you start by laying out the sheet called “Tables”. This sheet has entries for your teams and the dates that you play on. These table contain your master data for your league. Since they get referenced in the rest of the sheet this is where you would go if you want to use the correct a misspelled team name. This took me about a hour to setup and needs to get done once at the start of the season. After that you shouldn’t ever have to touch it.

The next thing to do is to setup the sheet called “Enter Data Here”. This is where your week-to-week data entry goes. There’s an row for each team in your league and a column for each week that you play. That defines a box for each score. So if the Beachcomber Cafe team scores 11 points in Week 1 (June 26th) you put an 11 in the appropriate box. There’s a separate area for penalty points should you get a late score sheet or a late payment for a player. This took me  about another hour to setup. You would have to set this up once also during the season. Adding this weeks scores took less than 15 minutes. Adding scores has to get done each week but I expect that this work will become common so it should take the same effort as me, week to week, as anyone else.

There is a sheet called “Results for Display”. This is the sheet that you want to print out or convert to PDF and distribute to your members. It’s exactly like the sheet that you are currently using for this with the exception you do not enter the data here. This sheet references the data on the previous sheet: “Enter Data Here”. The only thing would want to do to this sheet is fix the formating to suite your own league. I did address a pet peeve of mine in my rendition of this sheet. I show the team, then their total score, followed by their average. And the teams sort into places, first, second, third, by average. I’ve talked to a bunch of people about this while they agree with my math they say that sorting the scores by average rather than raw points is beyond the math skills of the common dart player. You don’t have to do it this way.

This sheet took me a little less than a hour to setup for MSDL. It requires weekly maintenance to sort the teams into order. The sorting columns are hidden so the maintainer has to know how to show hidden columns, sort from Z-to-A, and then rehide the columns. All told this takes about a minute for me to do.

There is a sheet called “Results for Display”. This is the sheet that you want to print out or convert to PDF and distribute to your members. It’s exactly like the sheet that you are currently using for this with the exception you do not enter the data here. This sheet references the data on the previous sheet: “Enter Data Here”. The only thing would want to do to this sheet is fix the formating to suite your own league. I did address a pet peeve of mine in my rendition of this sheet. I show the team, then their total score, followed by their average. And the teams sort into places, first, second, third, by average. I’ve talked to a bunch of people about this while they agree with my math they say that sorting the scores by average rather than raw points is beyond the math skills of the common dart player. You don’t have to do it this way.

The final sheet is called export_for_website. It took me about a half hour to set that up but for someone else the effort is all done. This sheet formats the data into one big long table suitable for direct import into a database. My plan is to import this data using a combination of tools which will take the sheet, convert it into a data format (CSV or comma separated value), load it into a database table and then then have some database code (a stored procedure) which updates the website tables. The trigger to load the database could be a special email recipient or I could put together a web form with a box for uploading the spreadsheet. Either method should be pretty easy to work for a novice user. If you don’t understand any of that I will be glad to explain it to you. If you do understand that but don’t get the next piece that’s completely okay. For advanced users I could set things up so that you directly import the data if someone is using the full Microsoft office Suite including Access and ODBC. Similar functionality is available to people who use OpenOffice. *Plug* I’m using open office to create all this and the package is Free!

FreeBSD’s geom makes life easier.

FreeBSD’s geom is the missing link that I’ve been searching for. Geom is an abstraction for mass storage providers and consumers which really cleans up the mass storage layer in Unix. In geom your disk drive combined with it’s driver is a mass storage provider. The filesystem layer is a mass storage consumer. Geom provides the glue in between providers and consumers which allows greatly enhanced function. Encrypted or compressed storage can be easily built using this framework.

I’m using geom with the automounter, amd to revamp my use of removable media. If you use the new gnome framework gnome’s hald does this also but it requires the user to unmount the drive. I like using amd here. Even though there’s a slight performance penalty amd will automatically unmount the drive for you after a configured period of inactivity. I find that this is an effective way of getting around one of Unix’s quirks. Unix doesn’t react well with a piece of the filesystem just disappears which is what happens when you remove a mounted USB stick. Amd can be configured to unmount the stick 30 seconds after you’ve stopped using it. Doing this almost eliminated the resulting kernel panics and length fscks that I experienced when I first started using USB storage.

The automounter is a utility that was designed to mount storage into the filesystem on demand. It works by providing an NFS look alike storage system. You literally mount the automounter into a directory in the filesystem. The automounter is configured with a map that assigns different directories within it’s filesystem to different pieces of mounted storage. So if amd is providing the directory /Volumes and has a mapping for MyUsbStick when you try to get a directory listing of /Volumes/MyUsbStick, the automounter has all the information needed to do the mount and provide access to the filesystem beyond. So far that’s about even.

Without geom that was a nice enough setup. But when the kernel attaches your usb stick in FreeBSD it gives it a name like /dev/da0s2e or /dev/da1s1. It’s easy enough to figure out what this means. da0s2e is the “e” partition on the second slice of the first “SCSI” drive in the system. USB and firewire drives a psuedo SCSI drives in FreeBSD and in Linux because the SCSI protocol was flexible enough to serve as the model for mass storage. But what happens if you have two systems and one of them has a SCSI controller in it or if you have two USB sticks. SCSI drives are number in the order in which they appear on the system so for the man with two USB sticks the whether they are connected as Drive a is da0s2e and drive b is da1s2e or vice versa depends on the order in which you plug them in. It turns out that geom embraces the concept of a volume label and can create a shadow device based on that label.

To use this feature of geom load the kernel module geom_vol_ffs (Available in FreeBSD from 6.x onwards) then add a label to your FreeBSD filesystems using the tunefs command:

# tunefs -L “MyLabel” /dev/da0s2e

Then you should end up with a device entry for your disk called /dev/vol/MyLabel

Have fun

— Chris

FreeBSD/WPA

Since I got my Mac some of my FreeBSD projects have been languishing on the back burner. Two are important, getting an IPSEC tunnel using IKE between FreeBSD (racoon) and OpenBSD (isakmpd frontended by ipsecctl) and getting WPA going. A couple of months ago I replaced WEP with wpa in my home wifi setup. There’s no arguing that the security is better and on the Mac it’s drop dead simple. I never understood what was going on in FreeBSD I understand it now. WPA appears to be divided into two parts like IKE. One part runs on the client and another in the Wireless AP. FreeBSD includes a program called wpa_supplicant which manages the WPA key exchange for you. To handle this it also has to manage the wireless interface. The automatic setup is actually pretty easy. I found this which helped me out. I wanted to understand what was going on under the hood. It turns out the setting up the config per the original article is the first step. Then run:

wpa_supplicant -B -Dbsd -iath0

as root. This handles the WPA negotiation. When ifconfig reports that you are connected you can run dhclient ath0 to connect.