In a parallel with the acronym LAMP, the overall stack is probably best described as “LAPD”:

• Linux
• Apache
• PostgreSQL
• Django

although there are refinements at almost every level, which I’ll go into below.

I18n with Django was a joy to implement. We had to have the site translated in six languages, but this was practically a doddle with Django’s core internationalization behaviour. Before we’d put the translations in, I clicked on “POLSKI” and was sure the l10n wasn’t working, until I spotted that the “ENGLISH” link had magically changed to “ANGIELSKI”.

The only issue we’ve had—which we’re hoping to work around—was deciding on an initial translation when the visitor arrived for the first time. Some parts of the system seemed to need l10n—the plumping for a specific version of the site—before Django’s LocaleMiddleware had done that for us, so we’ve had to force English until the user states otherwise. I’m hoping that a version 2 rewrite will fix that.

PostgreSQL might seem an odd choice of database to the LAMP community, who are used to MySQL’s minuscule overhead and often actively work around its deficiencies to keep that low overhead. But PostgreSQL’s stability and maturity outweigh the performance issues which—with a little care and attention—it’s possible to at least partly mitigate.

Compared to MySQL or Oracle, PostgreSQL also has the accolade of being supported by Django but having no outstanding database issues. Whether that’s because nobody else uses it, or because the integration is tight and relatively bug-free, I daren’t comment, but we’ve had nothing but seamless, transparent behaviour thus far.

Our Django-oriented hosting is with Webfaction, who are geared up for one-click mod_python Django deployments. The Django app sits in its own Apache process, while static files are served by the Linux server’s main Apache process: differential serving of content lets us take advantage of differential hosting fees. The hosting is a pretty good package, although TimeToLead.eu (despite being a pretty small application) is already finding the maximum memory package a bit restrictive, so we’ll need to keep an eye on that.

On top of this, Django has to a greater or lesser extent enabled us to use a whole host of other neat little technologies to integrate the site and improve both the user and the developer’s experience:

• The main Flash widget is written in Flex: I’m no particular fan of Flash myself, but it fulfills the remit admirably here. Because of text flow issues, there are actually six versions of the Flash file, one for each language, but the majority of the explanatory text is then picked up from an XML feed provided by Django, so can be retranslated by the client without recompiling Flex.
• Swfobject serves up our main widget and the YouTube file. It degrades well, but bear in mind that the API to it has changed considerably in the most recent major version.
• Unit tests in Django cover a number of stress points in the code (although not all, owing to the timescale). Towards the end, we were testing any major functionality changes by writing the tests first, which I was really pleased with.
• BeautifulSoup, which I mentioned a few days ago, is employed in the unit tests to check that certain content is coming through on the front end. It also managed to help us proof against a browser-dependent FOUC we encountered during development.
• Translations are managed by the Django application Rosetta: the most recent stable version is for Django 0.96 but the bleeding-edge repository copy seems to work OK on 1.0b1

Django has been the saviour here, living up to its promise as a framework for rapid application development. No framework is ever perfect, and there’ll always be arguments over the right way to proceed, reconciling what the framework developers want you to do with what the end developer wants to do, but this particular brief sojourn alongside Holovaty, Kaplan-Moss, Willison et al has been tremendous fun.

Web architects must understand that resources are just consistent mappings from an identifier to some set of views on server-side state. If one view doesn’t suit your needs, then feel free to create a different resource that provides a better view (for any definition of “better”). These views need not have anything to do with how the information is stored on the server, or even what kind of state it ultimately reflects. It just needs to be understandable (and actionable) by the recipient.

— Roy Fielding on creating new resources for REST architectures.

What if you want the same output for several different data structures? Let’s say we have three people, and we’d like to run the following include, saved as name.html, on all three:

<p>{{ person.firstname }} {{ person.surname }}</p>

Ideally you’d have the three people in a structure called people, and be able to run:

{% for person in people %}{% include “name.html” %}{% endfor %}

But what if the order matters? What if you need account_director to do different things in the template from project_manager? You’d want to send them through as separate data structures, and this:

{% include “name.html” %}
{% include “name.html” %}
{% include “name.html” %}

would just do nothing, because it references person and not account_director. An alternative is to turn name.html into a custom tag, taking arguments; but if the content of name.html were any more HTML-heavy then you might want to abstract it into the include nonetheless, and then you’d just be running a custom tag method, to pass variables to a template fragment, to render the fragment, to pass it back up to the template. It’d be nicer if we could do this with core functionality instead.

Using the latest development version of Django, we can implement basic variable assignment—getting person to reference whatever we want in the template layer—using the with template tag. This means that we can wrap each include tag to produce the required output:

{% with account_director as person %}
  {% include “name.html” %}
{% endwith %}
{% with project_manager as person %}
  {% include “name.html” %}
{% endwith %}
{% with lead_developer as person %}
  {% include “name.html” %}
{% endwith %}

It seems like overkill for our name.html example, but there are plenty of situations—I just found one, hence this post—where it makes sense to factor out code into an include: being able to fiddle with the template context before including it is a lightweight alternative to template tags and a tidy alternative to dumping the HTML right there in the primary template.

Django has its own library for building forms, currently called newforms (to distinguish it from the old library, deprecated but left around for compatibility reasons). The ModelForm class in this library interrogates a particular class from the model layer and builds an HTML form—complete with post-submit validation, field widgets, the lot. The beauty of this is that you can run:

import django.newforms as forms
from myproject.models import MyObject

class MyForm(forms.ModelForm):
    class Meta:
        model = MyObject

and that’s it: you can then use an instance of MyForm anywhere in your view layer to add or edit MyObject instances in the database.

Problems arise, of course, when the automatically generated form doesn’t provide exactly what you want. It might, for example produce select dropdowns for you with thousands of items, whereas with better knowledge of the page your user is on then you might be able to whittle that list down to a handful of items.

Let’s imagine your objects can belong to Django users i.e. in your models.py you have:

class MyObject(models.Model):
    owner = models.ForeignKey(User)

This means that when you try to create an object, the HTML from newforms will give you a select dropdown of all the users in the database. But what if you have loads of inactive users, cluttering up the dropdown? How do you get rid of them with the minimal amount of coding i.e. without having to build your own form if possible?

Luckily, newforms sticks the queryset it creates somewhere that you can fiddle with it. With lazy evaluation, you get a chance to change the queryset before it hits the database; with chaining, you don’t have to worry about rebuilding the queryset yourself, but can just add to it instead.

In the MyForm class, you can override the inherited __init__ method as follows:

def __init__(self, *a, **kw):
    forms.ModelForm.__init__(self, *a, **kw)
    self.fields['owner'].queryset = self.fields['owner'].queryset.filter(is_active=True)

This calls the existing superclass constructor first, passing along any values. Then, after the queryset has been created for the select dropdown but before it’s been evaluated, the queryset is modified to whittle down the resultant set of users to only those where the “is_active” flag is set to True.

Laziness and chaining make Django’s ORM layer very effective; forms and newforms help to greatly reduce the amount of coding required by the view and control layers; together they can really simplify a codebase.

But the eyes of even the most hardened LAMPer would widen, if he were to dip into a conversation to find someone saying the following about something they’ve built over PostgreSQL:

Yeah, I could add the current date/time to the end, so as long as you don’t add more than one [entry] with the same name within 1 second, it’ll be fine.

It might take you a while to work out (a) what that’s referring to (b) what kind of error in thinking it demonstrates and (c) what it implies about the quality of the underlying code. By that time—if you’ve worked at all with that sort of programming—your head may well be in your hands.

Matthew Somerville is well known for his accessible takes on rubbish websites: most useful is Traintimes, a layer on top of one of the equally poor commercial British rail sites; most notorious is Accessible Odeon, a fixing of the Odeon Cinema’s website that put their own substandard development to shame so much that in 2004 they had Matthew’s version taken down with legal threats. Remember: just because you’re successful doesn’t mean you’re not stupid.

Recently, Matthew gave an OGN talk on split tickets. These are train journeys which one cross-country train company sells at exhorbitant rates, whereas the components of the journey can be bought separately from the local companies for considerably less. This is what we infer; the rail websites try to keep it all quiet, in the hope that you might lose interest and stump up. The whole headshakeworthy situation is either a stinging indictment of the stupidity of privatizing a rail network and making passengers jump through ridiculous hoops to glean even the tiniest advantage, or a perfect demonstration of how the choice-enriched consumer can leverage capitalism in action: take your pick.

To come to the point: Matthew has set up a wiki where people have been adding the split tickets they’ve worked out in an ad-hoc fashion. Last Sunday I added K’s astonishing 47% saving for Oxford–Cardiff to the end of it, and there’s been little activity since. The page hasn’t got fantastic Google juice—”split tickets” means too many other things—and is just non-dynamic HTML, editable through the wiki but otherwise searchable only by eye.

At the moment, of course, there’s no reason for Matthew to expend any more effort on such a small and barely popular data set. But last Sunday I had a sudden instinctive jolt, to the effect that: more people would be likely to take advantage of split tickets if there was an easier way of looking up other people’s discoveries (my colleagues were recently trying to find split tickets and the wiki page was harder to use than Traintimes, for that very purpose).

With this in mind, here’s my take on accessificating the wiki page: Spliticket. It accesses a cached version of the page, and then using Python’s HTMLParser to hack away at the HTML it returns either a HTML or XML representation of any journey it finds. The idea is that it’s a bit easier to use on your mobile device, and lets you pin down journeys better. I’ve also included an option for searching strictly for the same journey: I’m not sure if split tickets—even returns—always work when the journey is reversed.

Spliticket also supports friendly URLs (inspired by Traintimes itself), so York to Edinburgh in HTML becomes <http://www.jpstacey.info/applications/spliticket/html/york/edinburgh/> and the aforementioned Oxford to Cardiff route in XML becomes: <http://www.jpstacey.info/applications/spliticket/xml/oxford/cardiff/> .

Mostly this was just something to fill idle hours, and also to convince myself of some ideas I’ve had recently about loose coupling, data reuse and open data. Hopefully at the same time (a) Matthew won’t take it as a dig, (b) others might find some casual use for it see it, and (c) a select few might see it as a testament to the ease of freeing information from solid if unsemantic markup. And maybe in six months’ time we’ll all be booking split tickets as a matter of course; by then, of course, Spliticket will probably be gratefully obsolete, replaced by the fully-fledged application that poor, embittered rail passengers deserve. I’m sure Matthew will build it if so.

Drupal has a core paging module that sorts out links along the lines of:

[« first] [< previous] [1] [2] [3] … [next >] [last »]

Great. But if, like me, you use the PHP $_REQUEST scope to stick lots of useful internal variables in (e.g. theme configuration, for in case you ever want to move your theme’s javascript/ directory in the future and don’t want it hardwired everywhere) then the pager code goes completely mad. Instead of links like:

http://…&page=2

It adds all the $_REQUEST variables onto the link too, creating:

http://…&page=2&sekrit_variable[]=quux&something_else=bar&blithely_generating_long_links=true

This isn’t a bug; it’s a feature. I’ve checked in theme_pager() and it does this on purpose.

So how to pare back the pager code, so it can still be reused if you want to also get some sort of use out of your $_REQUEST scope? Well, below is a snippet of code which in one sense overrides the default pager theme; but in another sense it just wraps around the core pager code. This is preferable to stubbing out the whole of theme_pager() yourself and then fiddling with it, because you’re introducing fewer maintenance issues by avoiding repetition of 95% the same code.

The principle is for phptemplate_pager() to swap everything briefly out of the $_REQUEST scope, run core theme_pager(), save the returned content somewhere, swap everything back into $_REQUEST and return the core-generated pager HTML as if nothing happened. Presto changeo:

/**
 * Implementation of theme_pager
 *
 * Prevents $_REQUEST[] madness
 * @ingroup themeable
 */
function phptemplate_pager($tags = array(), $limit = 10, $element = 0, $parameters = array()) {
  $temp_request = array();

  // Transfer $_REQUEST variables into array and delete from $_REQUEST
  foreach($_REQUEST as $rk => $rv) {
    $temp_request[$rk] = $rv;
    unset($_REQUEST[$rk]);
  }

  // Run theme_pager as before
  $ret = theme_pager($tags, $limit, $element, $parameters);

  // Transfer variables back
  foreach($temp_request as $rk => $rv) {
    $_REQUEST[$rk] = $rv;
  }

  return $ret;
}

Feel free to use. I see that theme_pager() also does the same thing to the $_COOKIE scope. I know, I know. But you’re on your own with that one. Teach a man to fish, and he can fish for cookies.

What I like most about progressive enhancement is that it gives me a way of tackling the age-old problem of turning a design for a client into a workable, accessible page without going completely mental. Before, if I saw a dropdown on the designs without a submit button or some such, I’d think, well, the HTML won’t have a submit button either, so I need to build the Javascript at the same time and get it all to work at once. This led me into all sorts of tangles—each one on its own easy to extricate myself from—and at the end of it I would have a form that would work or look right without every layer of technology in place.

But if you’re thinking progressively at the back of your mind all the time, you can start with an unstyled, unscripted form that works, and then use subtle styling to get exactly where you want to be. Some onloaded Javascript can give you a body.js class with which to hide unwanted submit buttons. At the end, if nothing else, the form still submits when Javascript is turns off, and it looks OK because you checked at the outset.

Progressive enhancement, ironically for the amount of work it implies, actually places the least stress (and maintenance workload) on the developer, and makes them learn a thing or two about layering and modularity at the same time. And as Steve points out, in this model there is no degradation: just the near-hotswapping of a range of technologies on top of a sturdy markup language that has been around in various forms for over a decade.

I’ve never been a fan of gratuitous Flash, but I’ve recently agreed to implement a complex page design (using swfIR) to help a client reuse existing content. Portrait images of users need to look “normal” on an index page, but then be rotated to line up with a fake “polaroid” background at around 2.5 degrees on the user profile page. Rather than have the client upload at least two copies of every image, which seemed a bit mean if we could avoid it, swfIR could do the rotation for us.

Apart from an initial problem, swfIR worked very well, the Javascript interface to the Flash proving very easy to use and the resultant image quality fine for our purposes. There’s the standard background problem with freeware Flash plugins (the background of the bounding box of the plugin’s influence on the page cannot be transparent, and thus chips away at the background polaroid blank on Ubuntu/Firefox) but otherwise it integrated seamlessly into both the design of the site and the technology of our existing Javascript workflow. It also coexisted happily in the browser with the jQuery and MochiKit frameworks, as we’d already restricted the latter’s scope to just the window.MochiKit object: compare this with the declarations on the swfIR site that neither MooTools nor Prototype sit well with their system.

The initial problem I mention above was one of URL paths. The path to the .swf half of the swfIR pair is hard-coded into the object returned by the window.swfir() function. Not only that, but it’s a relative path: just "swfir.swf". When the visitor was viewing the page .../node/100, the Javascript was creating a Flash embedded object that effectively referenced .../node/100/swfir.swf; but then when the visitor navigated to the aggregated view at .../view/all_users, a completely different file was being referenced by the embed. This situation was hardly helped by many of these apparent directories actually not existing on the server at all, but being phantoms, handled by Drupal and Apache cleverness.

The fix for this was eventually fairly simple, but it did involve hacking the Javascript. In swfir.js I changed the function declaration (line 354) to:

function swfir(path)

I then permitted this path to be brought in at the front of the existing hardwired path (line 364):

this.src = (typeof path == “undefined” ? “” : path) + “swfir.swf”;

I could in principle have just overridden it entirely and dropped the hardwired bit, but decided against it for backwards-compatibility reasons. On reflection, too, the object’s property this.src could probably have been modified post-instantiation, but I have enough JS headaches already, working out what different browsers consider to be “mutability,” to want to add another to the contents of my weary cranium.

That’s it, then: the full fix. I’ll mention the issue to the swfIR people shortly. A more foresighted solution would cause that function to accept a whole object as a parameter, an object whose properties could then be used to override any one of the configuration variables currently hardwired around line 360. But this quick and dirty change worked fine. I just wish I could show off the results, but the site itself is still embargoed. As soon as it appears in the real world then you can guarantee that I’ll point everyone I know at it, as it’s arguably one of the best, completest, tightest websites I’ve done, thanks in no small part to the efforts of my co-workers in design and additional development.

Among all this Flash hysteria, of course, it’s important to remember the wise words of Hugh Winkler (via the otherwise ever-quotable Sean McGrath)

Users have voted with their mice, and they’ve voted for the web experience — exploring the web information space using hyperlinks — as far more important than whizzy UI. Ask eBay. Ask MySpace…. It is good to have super expressive widgets — hear hear. But if you’re not pushing a bunch of hypertext down to my browser, you’re not helping me explore the space.

The lesson, I suppose is to use Flash, but use it sparingly, as a progressive enhancement.

Drupal is certainly very interesting. Its notion of presentation is remarkable in that, at a certain level, all content consists of homogeneous nodes, whether that consists of uploaded files, images, blog posts, taxonomy categories, or embedded YouTube videos. In addition its API for templating, both as a library of functions and as a workflow that one can hook into, probably rivals WordPress in its scope and power. At the same time, though, the implicit homogeneity makes it hard to structure fundamentally heterogeneous sites; and the API hooks are very difficult to unravel: frequently you’ll want to get at a function some ten levels deep, and probably three of those levels can be overridden by your own code, but which, and how?

I want to mention more at a later date, to do Drupal justice, but suffice it to say for now that the complex hierarchy of the hook-in workflow is almost entirely opaque in PHP, a language that provides rather terse error reporting, without the function debug_print_backtrace(). Well worth a look if you’re debugging spaghetti-code, especially when all you can see is the White Screen of Death. Sprinkle it around as the gentle British rain from heaven: lightly, but often.