More stuff about Postgresql that should be obvious.

I’ve been scratching my head on this one for far too long. I have a query under Postgresql which retrieves the distance between too points given the knowledge of their zipcodes. This work because I have an incomplete table of mileages between arbitrary three digit zipcode pairs. Each time I use this table my queries take a long time and I could never understand why. It has to do with the type that postgresql assigns to computed text fields. I was doing something like this:

SELECT * FROM worklist INNER JOIN partial_zipcode_mileage ON SUBSTRING(worklist.origin_zipcode, 1, 3) = partial_zipcode_mileage.origin_partial_zipcode...

The issue here is the type of the expression: SUBSTRING(worklist.origin_zipcode, 1, 3) as compared to the type of the field partial_zipcode_mileage.origin_partial_zipcode. The latter is a SQL CHAR(3) since it will always hold 3 characters. Postgresql assignes the first expression a type of TEXT since it has know way to know how bit a field you actually want. This prevents the postgresql engine from using the index and this, my query takes along time. Substitute SUBSTRING(worklist.origin_zipcode, 1, 3)::char(3) in the statement and all is happy.

Why math is important.

While I was sleeping we seemed to forget how to do math. This guy was quoted a rate of 0.002 cents per KB to use his Verizon Wireless Data Card while roaming in Canada. When he got the bill they charged him 0.002 dollars/KB. His story is here. What makes it sad is that the verizon customer service people don’t understand the difference and continue to quote him the lower rate while insisting that the charge on the bill is correct. All of this would be a non-issue if the marketing weasels at Verizon would just fess up to the fact that their price for roaming data is $2.05 / MB.

Exception trapping in pl/pgsql.

Learned a new trick with postgresql stored procedures today. This will probably appear to be obvious but it’s new to me. You can do exception trapping in pl/pgsql by you can also ignore some errors. The form is:


BEGIN
DROP TABLE foo;
EXCEPTION
WHEN undefined_table THEN NULL;
WHEN OTHERS THEN
RAISE NOTICE, ('Notice: ' || SQLSTATE || ' - ' || SQLERRM);
END;

Most MTA’s should offer SSL/TLS

Most MTA’s should offer Opportunistic TLS by default

I think that the time has come for most SMTP MTA servers to offer STARTTLS session protection by default. I see two reasons for doing this. Firstly, it takes a short amount of extra time and a little more CPU horsepower and that’s a resource that spammers cannot control. Secondly, opportunistic TLS brings email security a little more in line with the security model that most users expect.

Spammers

The majority of spammers out there are relying on stealing CPU time on machines that they don’t own. I don’t see them moving to TLS at the client side anytime soon. On the other hand legitimate email senders usually aren’t sending mail in such bulk that the cost of encrypting the session would be an onorous penalty. The practical end result of this would be differentiation of mail at the inbox. We would get mail from servers that used TLS to encrypt the session and mail from servers that didn’t. Assuming that the MTA server flagged the mail on this axis by adding a header, the end result is a hook for a statistical spam filter to use.

Users expectation

The second advantage would be a little added security in email during the transport from client MTA to server MTA. If everyone adopted opportunistic TLS encryption of the wire then sending email would better approximate the users expectations for security. Compared to physical mail email without TLS is like sending a postcard. No one sends postcards where security is a requirement because it’s obvious that everyone between the point where you drop the mail in the postbox and  the point of delivery  can just read the mail. Most users don’t expect that this is the case with email right now.

The advantage of opportunisticaly encrypting the mail is that we have a situation that we can grow into. If some server doesn’t do TLS in transport the mail still gets delivered.

Greylisting + MS Exchange 2003

I’m using greylisting to filter spam. It works quite well. If you aren’t of the technique this is how it works. Greylisting filters spam by testing the RFC compliance of the server that is trying to send mail to you. RFCs 2821 and 821 describe the meat and potatoes of sending email on the internet. The RFCs both specify that the receiver may tell the sender to queue the message and retry later because the receiver is temporarily out of resources. Greylisting exploits this to sift spam from legitimate email because many Spam sending programs cannot queue mail. As a method of spam detection Greylisting is great because it takes almost no resources on the receiving side to filter. Other methods of filtering are not so resource friendly. I find that Greylisting is rejecting over my inbound 90% of the spam. I used to say that it did this with 0 false positives but after reading these two threads I’m not so sure:

Leave it to Microsoft to rain on the parade.

I’m not going to stop Greylisting. It’s just been too effective at spam removal for me to even consider going without. I’m also aware of several people who are using Exchange to contact me who have not run across this problem. For me the solution to this potential problem is to contact some of the people who I know that are running Exchange and see what their awareness is on this problem.

Beer run

Got a beer making session in today. There was really no excuse for not doing this earlier. I’ve had all the ingredients in the house for the better part of a year. The yeast was dated Feb 2006 so I’m not sure quite what to expect here. I started the yeast on Thursday morning and it looked ready to go on Saturday morning so I put it into a starter culture. It never really took off but it was producing enough bubble to tell me that it was alive. The boil was okay but I overestimated the amount of water that I needed to boil and didn’t realize that I had until after I sparged the grain. I was still over when it was time to add the malt extract and start the boil. I tried to save as much as possible in a pot but I think I lost a little of the malt. The boil bubbled over twice making a mess on the stove but that wasn’t so bad. The biggest mistake as that the clamp on the wort chiller loosened up a little and so the wort chiller was adding tap water to my beer water. I like to boil all the water going into the beer. The original gravity was between 1.059 and 1.061 and the recipe called for it to be 1.061 ~ 1.064 so I think I’m okay. Well see when it starts to bubble in the primary. I hope to transfer it from the primary to the secondary next Friday night.

Followup

I remeasured the O.G. at 1.062. I’m pretty psyched. The yeast has started working. I’m getting about one bubble in the airlock every 10 seconds.

It’s been a few weeks so the beer should be pretty aged out by now. I should transfer it to the keg and get it on tap in a day or so.

Templating in Joomla and WordPress

Vindaloo backgroundI haven’t been writing in a while. It’s not because I don’t love it, I’ve been ramping up on a couple of projects and quite frankly that’s what I’ll be doing in about 10 minutes.

Among other things I’ve been using the Gimp to generate a new template for Vindaloo.com. I need to give credit where it’s due here. The person to talk to about this is Al Gordon. Vindaloo will be moving to Joomla soon. I’m writing a white paper on spam prevention and a few other things. I want to have a common theme for both this site and my Joomla site so I need to learn templating for both Joomla and WordPress. For Joomla the best thing seems to be PhotoShop and Dreamweaver. for the Open Source Developer who is on a budget. You can trade more time for less money and use the Gimp and Nvu. Nvu has a Mambo/Joomla Template Generator which is okay. If you are using FreeBSD and nvu make sure to get the latest copy of the port. I submitted a patch a month or so ago to enable Cascades, nvu’s CSS editor which is okay. Software:

Tutorials:

FreeBSD NIS server + Linux NIS client

Today I installed Fedora on my Desktop machine and tried to get it to talk to my NIS/NFS Server. I’ve run a NIS/NFS for a long while. It saves me my hair. But it’s always been FreeBSD on both sides. That’s easy to do. The FreeBSD handbook explains it quite well. I’ve been dipping my toe into the Linux waters alot lately and I figured that the next thing to put together was a Linux Client for my NIS/NFS server. The NFS part is easy. I used the automounter (amd) and the configuration is the same for both Linux and FreeBSD but NIS is another story. For my setup FreeBSS NIS server, Linux NIS client you need this patch to thefile: /var/yp/Makefile.dist on your server. Rebuild the maps and make sure that you have the automounter working and you should be good to go.

Greylisting via Spamd

Spamd

After far too long I’ve finally setup spamd to greylist inbound mail into vindaloo.com. This is something that I should have done a while ago. Before spamd I used a simple filtering setup for email based on Spam Assassin and using SA’s Bayes filter. It works okay but I was never happy with the performance that it needed from my box. When I first started this I was able to handle all the mail for vindaloo.com on a SparcStation 20 running OpenBSD. That’s not really special since I typically have less than 5 users. Disk space concerns forced me to upgrade to a VA Linux 2200, still running OpenBSD. That’s been very good but I’m now running into the same problems that I’ve had before. If there is any holdup in the mail system then the mail server gets hammered while the MX boxes on the internet offload mail. It’s easy to figure out why this is. I just look at the count of messages in my spam and junkmail folders. Lets see 376 messages in spam and another 246 in junkmail. That’s about 3 days worth of mail. That’s right. Despite Javascript veiling and everything else I do I get over 150 spams per day. Or at least I did until I started running spamd!

Slave Cylinder

Well, a while ago the slave cylinder in the M3 went south again. The last slave cylinder was about a year old but for a car that I drive 6000 miles a year in that’s pretty short. I’ve been buying the parts from Turner Motorsports and they’ve been okay but I’ll admit to being a little pissed at having to do this again. When I was a kid I didn’t mind being on my back underneath a car as the blood and strength drained away from my arms. Nowadays, working under a car just brings me closer to the day that I buy a lift. Anyhow thanks to my neighbor I was able to get the thing installed and again I get to see how nice the M3 is. For the past three weeks I’ve been getting my BMW kick from my wife’s 325i  and after two hours of driving the M3 I just have to say that there is no comparison. That’s saying something as I own a rather old E36 M3 and my wife has a very cherry E46 325i.