The Occasional Occurence

So we have some older Python code at work that uses old-style classes. We usually try and bring those up to date when we encounter them.

The other day one of the developers did that and one of our tests started failing. A simple change from:

class Foo:
    # stuff here

to:

class Foo(object):
    # stuff here

was all that happened.

Here is some code that encapsulates the problem and runs with some interesting results:

"""Two nearly identical classes with quite different behavior."""

class LameContainerOld:
    def __init__(self):
        self._items = {'bar':'test'}

    def __getitem__(self, name):
        return self._items[name]

    def __getattr__(self, attr):
        return getattr(self._items, attr)

class LameContainerNew(object):
    def __init__(self):
        self._items = {'bar':'test'}

    def __getitem__(self, name):
        return self._items[name]

    def __getattr__(self, attr):
        return getattr(self._items, attr)

if __name__ == '__main__':
    for cls in [LameContainerOld, LameContainerNew]:
        container = cls()
        print "Testing", cls
        try:
            'foo' in container
        except Exception, e:
            print "\tMembership in %s raised %r!" % (container.__class__, e)
        else:
            print "\tMembership in %s worked!" % container.__class__
        print "\t%s" % container.__getitem__

Here is some output from running that:

Testing __main__.LameContainerOld
        Membership in __main__.LameContainerOld worked!
            bound method LameContainerOld.__getitem__ of {'bar': 'test'}
Testing     class '__main__.LameContainerNew'
        Membership in     class '__main__.LameContainerNew'     raised KeyError(0,)!
            bound method LameContainerNew.__getitem__ of     __main__.LameContainerNew object at 0xb7d1c1ac

From what I can tell, when a membership test happens on the old-style instance, a membership test is done on self._items and it returns False. When the membership test happens on the new-style instance, it tries to treat it like a sequence and calls the __getitem__ method with index 0.

Does that seem like a correct analysis? Does anyone know why the behavior is different there?

Also look at the output for printing container.__getitem__. Isn't __getattr__ only supposed to be called when the attribute isn't present on the instance? Why does it return the __getitem__ method of self._items for the old-style instance then?

Very puzzling.

cw

The most basic case is very simple.

import time
import cherrypy

class WebSvc(object):
    @cherrypy.tools.caching(delay=300)
    @cherrypy.expose
    def quadruple(self, number):
        time.sleep(1) # make the real call somewhat costly
        return str(int(number) * 4)

cherrypy.quickstart(WebSvc())

That uses an in-memory cache and defaults to items expiring from the cache in 300 seconds (5 minutes). If you want to tweak that setting or others you can configure the caching tool to your liking.

cw

This is in response to `a post that asks if setting up caching in other web frameworks is as easy as in <http://slightlynew.blogspot.com/2009/02/full-web-service-with-http-caching-in-7.html>`_Rails` Ruby with Sinatra <http://slightlynew.blogspot.com/2009/02/full-web-service-with-http-caching-in-7.html>`_.

Ok, this post is on a controversial issue, and I'm entitled to my own ridiculous opinion. Therefore ...

The fact that we have 3 major automakers in the US that our economy apparently hinges upon is a problem. From the news, the talk is that a failure of any of them is disaster.

The problem with that is stuff fails. It just does.

It sounds like each of the Big Three has become a single point of failure. We try to avoid these in the technology world - we plan for failure in hardware and software.

def oversimplification():
    try:
        return make_cars()
    except ForeignCompetition:
        return make_better_cars()

We're human. We make stuff that breaks. We are flawed and fail a lot. It seems we have some economic single points of failure.

So now we are in a bad place. It reminds me of a monopoly situation. I realize these companies are three separate entities, but it seems like the public is in a position similar to a monopoly - the entities in question are so large and all encompassing that we need them whether or not we want them.

I want a US auto industry.

But why, oh why, OH WHY does everything in America have to be so BIG?

Grow, grow, grow. More, more, more. I'm tired of it.

So here's my solution (the culmination of my own ridiculous opinion) - many smaller automakers that adhere to open standards. I don't know how something like that gets regulated or started or anything, but I think that would be a much healthier situation for our nation. Lean companies that are making cars that they would like to drive for people like them.

There would need to be some sort of growth-cap or production-cap too. Think of it as a salary cap for the auto industry. You can only get so big.

Man, that doesn't sound very capitalist-like. I'm so back and forth on that sort of thing. I guess for that to be fair there would need to be caps on all industries. Moving on ...

Come on now! - we regulate against monopolies - we should regulate against (loaded language alert) cancerous big-business that entraps our children and furry woodland friends! I kid ... sort of.

Cancerous cells grow out of control until they form a mass that harms the vital functioning of the body, right? Isn't this a real-life economic cancer? Companies that have grown out of control and are now threatening harm to the vital functioning of the economy?

I've been picking on the auto industry because they are the ones in deep doo right now and are getting the headlines. Know that I was even more incensed at the Wall Street situation, and even wrote my representatives in Congress, rather than just idly blogging about it.

So there you have it. My solution for the life, the universe and everything, or at least the economy. Stop trying to get so big. You're only going to crash harder.

So, now, in the words of Captain Jean Luc Picard: "Make it so" (because everything that gets blogged about magically happens).

cw

I stumbled across Web Sequence Diagrams the other day.

It's a web-based diagramming application that uses a simple syntax to generate UML diagrams. No clicking around and positioning little boxes all over the place - just type in text that describes the diagram, and voila, there it is. He even provides a number of styles that you can apply to the diagrams. All in all, very slick.

Here's an example of some code that describes a request to a database backed website.

Client->HTTP Server: GET /foo
HTTP Server-->Database: SQL query
Database-->HTTP Server: Query results
alt resource exists
  HTTP Server->Client: HTTP 200 OK
else not found
  HTTP Server->Client: HTTP 404 Not Found
end

Here is the resulting diagram:

I read his blog a bit and it sounds like he is using Python, PHP and the Cairo graphics library. I wish the source was available, but it is not. He does have an open HTTP API though.

I think this might be an interesting solution to the images-in-docstrings discussion on comp.lang.python the other day, but only for including diagrams, of course.

cw

So I've been programming with Python since 2001 and I've never had the need to do anything that the standard import system didn't provide - until this week. We are planning on a little code reorganization for a project at work in preparation for collaboration from more developers. I wrote a simple custom importer/loader that let's a developer write

from application.widgets import foobar

instead of the longer

from application.widgets.foobar.widget import foobar

and the class foobar winds up in globals().

It's not groundbreaking functionality but it actually does add a little clarity in our situation. The whole task it was made quite simple by the features introduced in PEP 302 (that document is a great reference). Now, before anyone suggests that we could have just pulled the classes in via a __init__.py in the application/components directory, note that some components might depend on others which have not been imported and thus their imports would fail.

Anyhow, like I said, it isn't groundbreaking, but the very fact that you can customize Python's import system is neat. I got to thinking about what other ways I could hack the import system, and came up with a little web importer. I'll post the code below, only because I think it is a clever trick, not that it is something to use in development of a Real Application.

"""
Stupid Python Trick - import modules over the web.
Author: Christian Wyglendowski
License: MIT (http://dowski.com/mit.txt)
"""

import httplib
import imp
import sys

def register_domain(name):
    WebImporter.registered_domains.add(name)
    parts = reversed(name.split('.'))
    whole = []
    for part in parts:
        whole.append(part)
        WebImporter.domain_modules.add(".".join(whole))

class WebImporter(object):
    domain_modules = set()
    registered_domains = set()

    def find_module(self, fullname, path=None):
        if fullname in self.domain_modules:
            return self
        if fullname.rsplit('.')[0] not in self.domain_modules:
            return None
        try:
            r = self._do_request(fullname, method="HEAD")
        except ValueError:
            return None
        else:
            r.close()
            if r.status == 200:
                return self
        return None

    def load_module(self, fullname):
        if fullname in sys.modules:
            return sys.modules[fullname]
        mod = imp.new_module(fullname)
        mod.__loader__ = self
        sys.modules[fullname] = mod
        if fullname not in self.domain_modules:
            url = "http://%s%s" % self._get_host_and_path(fullname)
            mod.__file__ = url
            r = self._do_request(fullname)
            code = r.read()
            assert r.status == 200
            exec code in mod.__dict__
        else:
            mod.__file__ = "[fake module %r]" % fullname
            mod.__path__ = []
        return mod

    def _do_request(self, fullname, method="GET"):
        host, path = self._get_host_and_path(fullname)
        c = httplib.HTTPConnection(host)
        c.request(method, path)
        return c.getresponse()

    def _get_host_and_path(self, fullname):
        tld, domain, rest = fullname.split('.', 2)
        path = "/%s.py" % rest.replace('.', '/')
        return ".".join([domain, tld]), path

sys.meta_path = [WebImporter()]

You can use it like so:

import webimport
webimport.register_domain('dowski.com')
from com.dowski import test

That would fetch and import http://dowski.com/test.py.

There may be other Python libraries out there that do this better - I couldn't find any with a quick Google search. I can think of a number of features would be needed for a serious implementation of something like this (caching, HTTP-AUTH, signatures, remote package support, etc). For now though I'm just throwing this out there because I think it is neat.

Anyone else doing neat tricks with the import hooks that Python exposes?

cw