NFS – Old habits die hard

Old habits and myths die hard. Conventional wisdom asserts that UDP is better because it has lower overhead; then conventional wisdom suggests that you tune the buffer sizes to improve performance. On the face of things that would seem to work but once the the write size exceeds the max packet size, NFS delivers the packet by using multiple packets. Sending multiple packets triggers the issue because dropping just one UDP packet means the whole buffer must be resent. Contrast with TCP: yes the packet header is larger so less data can be sent and yes the receiving side has to ack each packet. But: with TCP if a packet gets dropped, only that packet needs to be resent; with a modern TCP stack the kernel will constantly adjust the window size to make the best use the available bandwidth. In other words NFS over tcp will automatically tune the buffer sizes for the current conditions.

MySQL lovefest

Great, just discovered how easy it is to break things with mysql views and stored functions. It turns out that to create a view after a dump, mysql must create table temporarily for each view, then one by one drop the tables and create views in their place.  This presents two potential problems. 1. It’s possible to have a view with more columns than you have in a table. 2. Views can use stored functions to modify results but stored functions aren’t a part of the mysql dump process until after the view have been defined.

The solution to the problem may be: Create the database, Create all the stored procedures and functions, create the tables and views from mysqldump –no-data. Reload all the data. It’s not. It looks like the only way to do this is to use information schema to make a list of tables to dump. Follow this up with routines, and then follow this up with views.

bash / ksh / pdksh

Fo my new job I’ve decided to try not to be so old and crotchety and use bash without complaining rather the just changing my shell to pdksh. Today I needed to process options in a shell function which I’ve done in ksh before. It turns out that you have to preface your option processing with OPTIND=1 if you are in a function. Dunno why but I’ll find it out.

py2exe + anydbm error

So, I tried something simple yesterday adding a call to anydbm to a python program that I plan to distribute on windows with py2exe. Doing so I ran into this error:

ImportError: no dbm clone found; tried ['dbhash', 'gdbm', 'dbm',  'dumbdbm']

Turns out that the way py2exe works though it misses the dependency that anydbm has on a db module. The moral of the story is that if you want to use anydbm and py2exe you need to do something like:


import anydbm, dbhash

f = anydbm.open("dbname.db", "c")
...

Finally sat down with sqlalchemy…

I’ve been meaning to sit down and play with sqlalchemy for a while now. As an old person though my needs are a little different. Most people use the ORM model to relieve themselves of the burden of dealing with SQL database engines. This is all fine and dandy if you have the luxury of using an SQL database solely as a repository for data with permanence. However, SQL databases can do much more than that. In this vein the sqlalchemy tutorials don’t tell you much about database introspection, that is figuring out what the layout of a table is from the information available in the database. Introspection is very important to me because I frequently create tables (and contraints and triggers etc) on the database. I also have a frequent need to an SQL feature called views but that’s a story for a different day. I came up with this code:

#! /usr/bin/env python

''' ============================================================================================
Program: sqla.py -- A test of sqlalchemy's ability to introspect tables from a database.
============================================================================================ '''

from sqlalchemy import *
from sqlalchemy.orm import *

metadata = MetaData("mysql://scott:tiger@mysql.example.com/test")

workEntryMeta = Table('work_entry', metadata,
                      Column('work_entry_id', Integer, primary_key=True),
                      autoload=True, )

class WorkEntry(object):
    pass

def sqlToStr(c):
    if c is None:
        return "NULL"
    else:
        return str(c)

mapper(WorkEntry, workEntryMeta)

session = create_session()
q = session.query(WorkEntry)
entries = q.all()

headers = None
for e in entries:
    if headers is None:
        headers = [ c for c in e.__dict__.keys() if c[0] <> '_' ]
        print headers
        print "\t".join(headers)

    d = [ sqlToStr(e.__dict__[c]) for c in headers ]
    print "\t".join(d)

This creates an ORM object for the table work_entry in the current python program and grabs all the rows from the database printing each one as it goes.

Website setup.

This seems obvious but it’s really handy to have my website setup with DAV access to the backend for administration purposes. I’ve recently setup on of my sites this way and it works out quite nicely.