What if you want the same output for several different data structures? Let’s say we have three people, and we’d like to run the following include, saved as name.html, on all three:

<p>{{ person.firstname }} {{ person.surname }}</p>

Ideally you’d have the three people in a structure called people, and be able to run:

{% for person in people %}{% include “name.html” %}{% endfor %}

But what if the order matters? What if you need account_director to do different things in the template from project_manager? You’d want to send them through as separate data structures, and this:

{% include “name.html” %}
{% include “name.html” %}
{% include “name.html” %}

would just do nothing, because it references person and not account_director. An alternative is to turn name.html into a custom tag, taking arguments; but if the content of name.html were any more HTML-heavy then you might want to abstract it into the include nonetheless, and then you’d just be running a custom tag method, to pass variables to a template fragment, to render the fragment, to pass it back up to the template. It’d be nicer if we could do this with core functionality instead.

Using the latest development version of Django, we can implement basic variable assignment—getting person to reference whatever we want in the template layer—using the with template tag. This means that we can wrap each include tag to produce the required output:

{% with account_director as person %}
  {% include “name.html” %}
{% endwith %}
{% with project_manager as person %}
  {% include “name.html” %}
{% endwith %}
{% with lead_developer as person %}
  {% include “name.html” %}
{% endwith %}

It seems like overkill for our name.html example, but there are plenty of situations—I just found one, hence this post—where it makes sense to factor out code into an include: being able to fiddle with the template context before including it is a lightweight alternative to template tags and a tidy alternative to dumping the HTML right there in the primary template.

Django has its own library for building forms, currently called newforms (to distinguish it from the old library, deprecated but left around for compatibility reasons). The ModelForm class in this library interrogates a particular class from the model layer and builds an HTML form—complete with post-submit validation, field widgets, the lot. The beauty of this is that you can run:

import django.newforms as forms
from myproject.models import MyObject

class MyForm(forms.ModelForm):
    class Meta:
        model = MyObject

and that’s it: you can then use an instance of MyForm anywhere in your view layer to add or edit MyObject instances in the database.

Problems arise, of course, when the automatically generated form doesn’t provide exactly what you want. It might, for example produce select dropdowns for you with thousands of items, whereas with better knowledge of the page your user is on then you might be able to whittle that list down to a handful of items.

Let’s imagine your objects can belong to Django users i.e. in your models.py you have:

class MyObject(models.Model):
    owner = models.ForeignKey(User)

This means that when you try to create an object, the HTML from newforms will give you a select dropdown of all the users in the database. But what if you have loads of inactive users, cluttering up the dropdown? How do you get rid of them with the minimal amount of coding i.e. without having to build your own form if possible?

Luckily, newforms sticks the queryset it creates somewhere that you can fiddle with it. With lazy evaluation, you get a chance to change the queryset before it hits the database; with chaining, you don’t have to worry about rebuilding the queryset yourself, but can just add to it instead.

In the MyForm class, you can override the inherited __init__ method as follows:

def __init__(self, *a, **kw):
    forms.ModelForm.__init__(self, *a, **kw)
    self.fields['owner'].queryset = self.fields['owner'].queryset.filter(is_active=True)

This calls the existing superclass constructor first, passing along any values. Then, after the queryset has been created for the select dropdown but before it’s been evaluated, the queryset is modified to whittle down the resultant set of users to only those where the “is_active” flag is set to True.

Laziness and chaining make Django’s ORM layer very effective; forms and newforms help to greatly reduce the amount of coding required by the view and control layers; together they can really simplify a codebase.

But the eyes of even the most hardened LAMPer would widen, if he were to dip into a conversation to find someone saying the following about something they’ve built over PostgreSQL:

Yeah, I could add the current date/time to the end, so as long as you don’t add more than one [entry] with the same name within 1 second, it’ll be fine.

It might take you a while to work out (a) what that’s referring to (b) what kind of error in thinking it demonstrates and (c) what it implies about the quality of the underlying code. By that time—if you’ve worked at all with that sort of programming—your head may well be in your hands.

The comparison in the Guardian website is with Facebook (which is turning rapidly more socially evil by the minute anyway). I think it would be more illuminating to compare mySociety sites with such large, government-commissioned (and consultant-propelled) projects like the NHS Spine.

The Spine is a vast, overarching superproject; it’s a messy amalgam of large, penny-pinching businesses; separation of responsibilities—bizarrely—is in part geographical, so despite purporting to provide a single system there’s no guarantee (outside of the weird steampunk astrolabe brains of consultants) that Yorkshire and Humber will be able to chat freely with the West Midlands. It’s been put together without any real thought about whether the problems tackled are the ones that the problem-sufferers (NHS staff and patients) encounter most often. In a nutshell, i’s overblown, overpriced and over the top.

But if mySociety had been brought in? You would have around a dozen projects, all quite tiny, and all solving fairly simple, easily-scoped problems. Most importantly, they would all stem from consulting the staff: each would have a valid reason for existing before even the planning began. The basic version of each site would be ready within six weeks, and a more polished version in three months. There’d be ongoing updates and improvements as user requests made them appear worth while ((the Spine, on the other hand, would shudder at the thought that the hoi polloi of front-line employees might dictate project direction, rather than a consultancy firm). And from the point of view of an end result, almost everything would be ready by now, in comparison to a couple of fringe benefits: it would be improving staff and patient experience today, across the country, in those specifically targeted areas, by a hundred times the effect the Spine will have, and for a hundredth of the cost.

mySociety produces perfect examples of the open-source maxim: find an itch, and scratch it. But what distinguishes them from almost every other FOSS project (apart from perhaps Ubuntu) is that they’re happy to treat other people’s itches.

… This brings us to the second obviously-bad thing that can go wrong with code bases: copy and paste. It doesn’t take very long for programmers to learn this lesson the hard way. It’s not so much a rule you have to memorize as a scar you’re going to get whether you like it or not. Computers make copy-and-paste really easy, so every programmer falls into the trap once in a while. The lesson you eventually learn is that code always changes, always always always, and as soon as you have to change the same thing in N places, where N is more than 1, you’ll have earned your scar….

Code reuse is great, although some languages punish you for it: Coldfusion takes a performance hit on behalf of debug logging whenever it crosses a component boundary, even if debugging is turned off. And a lot of includes scattered across the place can make it difficult to track what’s going on (unless you use those very IDEs that Steve suggests treat code like heaps of dirt to be rearranged). I suppose ultimately it’s about developing an instinct for balancing short-term performance, ongoing maintainability and long-term scalability, and you only learn that by dint of many, many scars.

There’s already a tutorial with a CCK submodule available. The premise of the sample module is to provide two fairly generic numerical formats. In principle that simplifies the submodule, but in practice it also means it’s difficult to work out precisely why particular programmatic choices were made and what you can tweak. Also, at work we wanted a much richer content type. Here’s what we wanted to display:

1. Build an “Available in” field for e.g. all the different formats that a book might come in.
2. Record the dimensions DxWxH
3. And some dscriptive information
4. And let the user pick a thumbnail from a dropdown
5. And let all of this be done multiple times

Here’s a mockup of what we wanted to appear on the admin page (excuse styling—Wordpress isn’t the best for this):

And here’s sort of what we wanted on the front end (again, sorry for the HTML: it’s table-heavy as it’s a one off):

Available in
Hardback			Paperback			Trade p/bk
	ins	cm		ins	cm		ins	cm
W	5.1	13	W	5.1	13	W	5.1	13
H	7.6	19.2	H	7.6	19.2	H	7.6	19.2
D	0.79	2.0	D	0.79	2.0	D	0.79	2.0

It was difficult, from the simplistic submodule available, to work out exactly how you leverage complex editing widgets, and how you get CCK to save all the database information in the five columns we needed: description, thumbnail image reference and three dimensions in inches. What’s worse is that a CCK submodule with an incomplete set of hook functions seems to do a fandango on certain areas of the database (possibly the cacheing of certain bits of the module’s behaviour): certainly developing a CCK submodule was quite painful, as the process of trial and error frequently required a database restore so as to try again. That tutorial does warn that “all hooks are required. CCK expects them to all be present and will not function correctly if some are missing.” But there’s a world of developer pain in those brief phrases.

Since I originally mentioned the problems we were having on the development mailing lists, a couple of people have asked me how to build a CCK submodule to support rich fields. With this in mind my employers Torchbox have very gracefully let me make the module we built public, with some minor edits (the client’s site isn’t live yet, so names and details changed to protect the innocent).

The module is available here. It consists of 4 1/2 pairs of functions, as follows:

1. Declare:
   1. field types that the module supports
   2. widgets that the module supports, for editing fields on the node-edit page
2. Settings for:
   1. field (so settings inherent to the way that particular field is stored e.g. decimal accuracy for numeric values)
   2. widgets (so e.g. the image gallery that populates the dropdown for the wireframes)
3. Handlers for:
   1. field, so what happens to it when a node is loaded
   2. widget, so pre-processing of the field to fit a HTML form, or producing a chunk of Form API, or post-processing the HTML form into a database-saveable format.
4. BONUS UNNECESSARY FUNCTION: cck_book_complex_input just produces the Form API for function 3b. Ignore this one!
5. Formatters, which give you output options for fields on the “Display Fields” page.
   1. Declaring formatters
   2. Formatters

A few extra pointers:

1. CCK does most stuff for you, so you don’t need to actually load/save field contents to the database; declaring the columns for widgets and fields is enough.
2. If you’re in the developing phase with your CCK submodule, back up the database first. See above.
3. We sidestepped formatters altogether as they don’t cope with e.g. multiple field values that need a header value. Instead, we just overrode theme('field'). One thing, though: you do need some formatters present, or the admin configuration page for “Display Fields” defaults every dropdown to “<Hidden>”, and you can’t get anything to appear.
4. To reiterate, CCK needs all hooks present in its submodules from the start. The further you get from Drupal core, the more bleeding-edge APIs will get.

Note that as I say we had to peer into the node (use var_dump($node) and search for the right array) and then override the theme, because we wanted a fancy header to our field. But that was pretty painless—mostly a matter of knowing one’s PHP rather than one’s Drupal—and it all worked smoothly otherwise. And we ended up with by and large exactly what we wanted. It works and, in retrospect, Drupal has helped us to write readable, maintainable code.

The hallmark of a good frameworked site is that it’s not easy to guess which framework was used, and TeamSugar manages that admirably. While I can’t really comment on the design—I find most networking sites pretty busy, and this is no exception—it certainly feels like a lot of hard work is distancing the user experience from the limitations of the framework, and that’s admirable from a usability perspective. The fewer visitors jumping through hoops to indulge the whims of your CMS the better.

There are a couple of interesting observations in that thread, mostly because they’re guarded criticism coming from an otherwise happy user of the framework:

… We have four app servers now, I believe, and a dedicated database server. We have also extensively edited the code. The architecture of drupal, unfortunately, lends itself to doing things like issuing one query per module per node on a page. For our 8 content sites this isn’t too bad because they’re easily cached and don’t make use of many modules. For TeamSugar it’s much worse because each feature generally requires at least one module. We cache where we can and refactor code elsewhere…. [link to 1971521]

… When you’re creating web applications you’re typically going for one of three things: speed/scalability, flexibility, or maintainability. Drupal is high on flexibility, average on maintainability, and poor on speed, in my opinion. If your priority is maintainability then I’d go for one of the RAD environments like Rails or CakePHP. If speed is your priority then I’d write your own lightweight framework, or whatever, and do it yourself…. [link to 198497]

If you haven’t ever had cause to use it, there’s a standard called ETag out there which you can probably implement using existing technology that can boost the efficiency of your content delivery a hundredfold. Along with the longer-standing HTTP header Last-Modified it can be used with compliant browser/aggregator software to drastically lower your overheads, while scarcely impacting on your non-compliant users. And although the two standards see most use in the blogosphere, they can be used for anything else from a company’s record in a directory to enormous high-resolution image feeds from astronomy laboratories.

The idea is that you embed in every outgoing page request a couple of HTTP header lines. That’s easier for the total n00b than it sounds: you can do it in one line each with, say, the PHP function header() or the Coldfusion tag <cfheader>, or even in the HTML if you don’t have that level of access, using <meta http-equiv=.../>. The point is that you set the following two flags on the “envelope” that surrounds the page you send to the browser:

Last-Modified: Wed, 15 Nov 2006 18:20:54 +0000
ETag: “78c4d3d8-1834-11dc-8314-0800200c9a66″

Typically for the first field you’ll want e.g. the latest date from your RSS feed, or the date on which a semi-static page was last edited. The second field is really up to you: if you never go back and edit posts without changing the published date then (a) well done you—there’s a space in heaven already reserved—and (b) you can just calculate ETag from the published date. It can actually be the published date if you’re a bit slack, although you might want to hash it instead, in the way that you might with passwords, to avoid any sloppy client software depending on ETags being dates.

What happens next? Well, you won’t see anything at first. But compliant software that’s visited your site before will start sending you two headers that correspond to your original submissions:

If-Modified-Since: Wed, 15 Nov 2006 18:20:54 +0000
If-None-Match: “78c4d3d8-1834-11dc-8314-0800200c9a66″

The idea is that you make sure that you can compare these to the values you’re about to send out, quite early on in your workflow. That way, if they match, you can immediately terminate all further work and just send a “304: Not Modified” HTTP header. The result? Well, with quite complex pages, involving the computation of tag clouds and term hierarchies and archive structures, you can work out quite early on whether it’s worth bothering, or whether the remote client will know exactly what to do if you just tell it, concisely, that nothing has changed since it last looked.

Word to the wise, though: if you are going to send a 304, you should also send your two headers as you always would. If you don’t then you only win out every other time, because the remote client will see the absence of ETag and Last-Modified headers and duly forget the ones it had in its cache.

I wondered this as I got far enough into a PHP publishing system that I had to start thinking about adding new content through the system (rather than just jamming it into the database by hand, which is why the previous incarnation has sadly fallen into disuse). As it’s typically configured, PHP will add backslashes to anything it doesn’t trust: hence the comment “it’s a great site you’ve got here” will, when submitted by a POST request, become “it’s a great site you’ve got here”. Whether or not your server does this automatically can be checked by calling the function magic_quotes_gpc() (I realised only the other day that “gpc” stood for “GET, POST and cookies:” I probably have some catching up to do). In performing this blanket adding of slashes, PHP prevents the unwary coder from leaving his site open to both unintentional database hiccups and intentional malevolent attacks, the SQL injection attack.

All well and good, but my application is heavily object-oriented. Such objects store whatever content you give them, as well as optionally writing it to the database. If I want these objects to persist (even for the course of a single request) then any access to their internal storage must yield sensible data: those slashes have to disappear before the articles appear in an RSS feed, or on the website itself. So when the CGI environment gives slash-added content to an object, the object needs to know to both add it to the database verbatim and to produce it for viewing with the slashes removed. It can either do this by storing it in a slash-removed state or by placing a filter on its outputs.

There’s a further complication, in that content can also be written to an object by the PHP application itself: the publishing of all my unpublished articles, for example, would change the status of their accompanying objects without reference to any CGI variable. If I assumed all of this content had had its slashes escaped, then this article, for example, would lose all of its ’ text, because the object would assume they’d been added by PHP’s internals: in my second paragraph, the “after” string would look like the “before” string, and the “before” string would instead break the database insertion. In addition, what if the server is reconfigured? Can I trust my hosting company to never change the configuration of PHP, even accidentally during an upgrade?

I found myself lost in a maze of adding, removing and then adding slashes, with no clear way of deciding. Suddenly I decided: why not use one of PHP’s major downsides—that it doesn’t support persistence of objects from one request to the next very well, and hence each action is fighting against the overhead of constantly recreating and recompiling code—to ascertain which input/output processes were the most frequent (and most public) and hence needed to be the fastest? I drew a flowchart of a typical object’s behaviour and, by identifying which channels could be safely bottlenecked, arrived at a reasonable solution to the problem.

From my phrasing it’s clear that it was a foregone conclusion: I wanted, more than anything else, for content to flow straight from the database (through the object if applicable) to the user. This content needed to stay in any object in a simple, de-slashed form, so it could flow and flow as long as the object was in existence. That meant that incoming CGI content could not be stored with its added slashes intact. Counter-intuitively, then, my solution was to undo PHP’s default safety mechanisms, unescaping the CGI content and storing it raw, and then without fail adding slashes to anything that CGI or my application wanted to add to the database. This would be my bottleneck: everything else would be as fast as it could be.

Exit gracefully: ensuring all incoming content can be added to the database safely is not necessarily the most efficient or desirable long-term solution. By examining the likely workflows for content, it’s possible to make pragmatic decisions on where content should be pre-processed and where it should be left alone. Consider all your overheads, including that of short-term programming and long-term cumulative processing time: this will vary depending on your environment. Also, if you’re aware of a safety net, over the presence of which you have minimal control, account for the possibility that someone might one day remove it.

foo
foo bar
quux
blort wuu spong

to the expanded form:

foo
foo
bar
quux
blort
wuu
spong

efficiently and cleanly, only using SQL? (In case anyone’s worried, I’ve scrubbed the data sets of any personal details they might have previously contained: any resemblance to the real Blort Wuu-Spong is entirely coincidental.)

I finally decided it wasn’t possible, and although without the pure mathematics to back me up I could have kept hunting—partial solutions involving a self-join for each whitespace splitting kept rearing their heads—what finally convinced me was comparing the behaviour of SQL with that of XSL(T). The two are more alike than you might think; and no, I don’t mean SQL and XQuery, although that easy comparison provides a clue for the underlying similarity.

In XSL(T), the XML node in your original document(s) is in a sense king: it’s considered bad form (and is at any rate inefficient) to do data management on some transient data set, created within the template. Loops work best over nodesets rather than with some sort of conditional or from/to structure. This stems from XSL(T)’s underlying functional paradigm, where each nodeset is created

Of course, it’s always possible to twist non-functional behaviour out of the stylesheet (and most real-world solutions have to take a pragmatic approach to such programmatic purity) and interpreter-specific kluges exist to node-ize strings based on some non-XML token, but the language works fastest and cleanest when it’s hanging functions off nodes.

In SQL, the equivalent to the node in an XML document is the row in a query. Rows are passed around, compared with other rows based on the content of some of their cells, tied together and discarded, but very rarely can rows be created out of thin air. The closest one gets is the LEFT/RIGHT OUTER JOIN where the ON-condition is not satisfied: then the left-hand row, rather than being discarded as in the INNER JOIN, is in a sense tied to a row of NULLs. Although that equates to it being tied to no row at all, then when the SQL99 dust settles and post-processing can begin, NULLs can be reinterpreted (Coldfusion does this without being asked, for example).

So to create new rows, one can UNION two rowsets, or entangle the rowsets with some sort of a JOIN, but in simplest, non-iterative SQL, there ought to be no easy way to make one row magically split into two, or maybe three, or maybe four, based on its textual content. It breaks the underlying principle, that rows should flow through the SQL into bit-buckets or the STDOUT tray, but shouldn’t be tossed into the stream with flamboyant verve like chillis into a stir-fry.

Exit gracefully: regardless of the data itself, the data model that a given language’s designers had in mind can have the most effect on what’s plausible to do in the language. Almost all languages evolve through proprietary extensions until they can do associative arrays, every kind of loop structure and, if left alone for long enough, GOTOs, but being able to complete a task with a given language is not the same as being able to complete it, for a sufficiently large data set, before the death of your server, your development team or the universe.