In a parallel with the acronym LAMP, the overall stack is probably best described as “LAPD”:

• Linux
• Apache
• PostgreSQL
• Django

although there are refinements at almost every level, which I’ll go into below.

I18n with Django was a joy to implement. We had to have the site translated in six languages, but this was practically a doddle with Django’s core internationalization behaviour. Before we’d put the translations in, I clicked on “POLSKI” and was sure the l10n wasn’t working, until I spotted that the “ENGLISH” link had magically changed to “ANGIELSKI”.

The only issue we’ve had—which we’re hoping to work around—was deciding on an initial translation when the visitor arrived for the first time. Some parts of the system seemed to need l10n—the plumping for a specific version of the site—before Django’s LocaleMiddleware had done that for us, so we’ve had to force English until the user states otherwise. I’m hoping that a version 2 rewrite will fix that.

PostgreSQL might seem an odd choice of database to the LAMP community, who are used to MySQL’s minuscule overhead and often actively work around its deficiencies to keep that low overhead. But PostgreSQL’s stability and maturity outweigh the performance issues which—with a little care and attention—it’s possible to at least partly mitigate.

Compared to MySQL or Oracle, PostgreSQL also has the accolade of being supported by Django but having no outstanding database issues. Whether that’s because nobody else uses it, or because the integration is tight and relatively bug-free, I daren’t comment, but we’ve had nothing but seamless, transparent behaviour thus far.

Our Django-oriented hosting is with Webfaction, who are geared up for one-click mod_python Django deployments. The Django app sits in its own Apache process, while static files are served by the Linux server’s main Apache process: differential serving of content lets us take advantage of differential hosting fees. The hosting is a pretty good package, although TimeToLead.eu (despite being a pretty small application) is already finding the maximum memory package a bit restrictive, so we’ll need to keep an eye on that.

On top of this, Django has to a greater or lesser extent enabled us to use a whole host of other neat little technologies to integrate the site and improve both the user and the developer’s experience:

• The main Flash widget is written in Flex: I’m no particular fan of Flash myself, but it fulfills the remit admirably here. Because of text flow issues, there are actually six versions of the Flash file, one for each language, but the majority of the explanatory text is then picked up from an XML feed provided by Django, so can be retranslated by the client without recompiling Flex.
• Swfobject serves up our main widget and the YouTube file. It degrades well, but bear in mind that the API to it has changed considerably in the most recent major version.
• Unit tests in Django cover a number of stress points in the code (although not all, owing to the timescale). Towards the end, we were testing any major functionality changes by writing the tests first, which I was really pleased with.
• BeautifulSoup, which I mentioned a few days ago, is employed in the unit tests to check that certain content is coming through on the front end. It also managed to help us proof against a browser-dependent FOUC we encountered during development.
• Translations are managed by the Django application Rosetta: the most recent stable version is for Django 0.96 but the bleeding-edge repository copy seems to work OK on 1.0b1

Django has been the saviour here, living up to its promise as a framework for rapid application development. No framework is ever perfect, and there’ll always be arguments over the right way to proceed, reconciling what the framework developers want you to do with what the end developer wants to do, but this particular brief sojourn alongside Holovaty, Kaplan-Moss, Willison et al has been tremendous fun.

At the time I built the HTML parser using core SAX-based HTML parsing, and it was horrible. SAX works in a basic sense, but you have to build your own internal state engine, track which elements have gone past while working out what to do with the current context, and even write rules for what to do when the underlying dumb parser encounters HTML entities: no mean feat when the document is peppered with – en dashes.

Not only was writing the rules initially a pain in the rear, but adding new rules and bugfixing the existing ones was even worse. But I lived with SAX, because I was deploying on shared hosting: I presumed that this was the best option available if I couldn’t install any new shared libraries.

Not true! I’ve just rebuilt the entire parsing layer with Beautiful Soup, a Python HTML/XML parser library which (a) is available as a single file and (b) works out a decent HTML DOM tree from pretty much anything you throw at it.

Try it yourself, if you have to do any HTML parsing.It’s astonishing; beautiful, in fact. I will never write another SAX parser ever again, which I’m sure I’ve said before.

To this end, four major environmental NGOs (Greenpeace, Friends of the Earth, the World Wide Fund For Nature and CAN Europe) have put together the following site, to help ordinary people—EU constituents—compel their representative MEPs to pass a package of laws that will tackle climate change and hopefully help to save the planet:

(Disclaimer: we built the site. More on the tech later: go badger your MEP first!)

[Edit: changed timescales on request]

This model of publishing in part explains the thinking behind his recent post, robustly titled Business Requirements are Bullshit. But Steve’s audience—compared to the web at large—is a small, self-selecting group. So although keeping in mind some of the details of his recent post can prevent you investing in dead-end projects, I just can’t agree with the overall conclusions about software development.

Although it’d be nice for everyone in the world to have programming skills, and to be able to behave like autonomous itch-scratching units, that simply isn’t the case. The vast majority of people need software built for them, and software builders are a demographic, with a broad range of shared interests and a vast landscape of shared uninterests. What if you can’t program but you want some software? Do you just sit there, or do you pay someone to build software for you? Should that person in turn refuse the money, saying “that’s too risky: Steve said so?”

Steve’s rant is aimed at CEOs instead, but the principle still stands. To what extent to people have to clamour for a particular feature before a CEO will say “well, I don’t want that, but I do want your money?” Personal phone calls? Petitions to their local MPs? Pre-ordering? Pressing themselves against the windows of electronics shops and drooling on the glass? Demanding it be available on the welfare state? Well, the canny software house would have started building the software before any of the above had happened. But how could they know, if they’re not secretly telepathic? Well, among other methods, by gathering business requirements.

From Steve’s point of view it’s less of an issue, because he works at Google, the House of Blue Sky Development. And I don’t begrudge him that privileged viewpoint at all, because Google has earned the business success that bankrolls schemes like twenty-percent time projects. If every business had oodles of cash to throw at developers as they wander off along tangential projects, then maybe none of them would need to through half an oodle at a discovery phase. But as long as money is a locally limited and unevenly distributed resource, then there will need to be different solutions to the problem of working out what to actually build.

The ultimate direction of software development proposed by Steve is just far too exclusive, and it’s been the bane of open-source projects for years. Project after project caters for its tiny community, never reaching out to what other communities might need; they start to cool off, then founder; the codebase is mothballed, and the project finally expires. Worse, in Steve’s world, we’d have to wait for blind people and RSI sufferers to write their own FOSS voice recognition software: maybe the blind person could hold the keyboard and mouse, while the RSI sufferer tells them what’s on the screen. Meanwhile, able-bodied programmers develop that sort of stuff for Vista without a qualm, and they can charge the earth for it because people want it.

Ubuntu’s recent successes might be almost entirely attributed to the fact that (a) the project is well managed and directed and (b) they reach outside their own community, and solve problems for people other than computer programmers. Despite Ubuntu, Gartner recently announced that Linux had a 4% market share. 96% of consumers opt for software built by people who on the whole weren’t solving their own problems.

The first commenter on Steve’s recent post says:

I find that a lot of Free Software is awful for exactly this reason — the authors built it for themselves. Their software only works for other hardcore programmers because they can put up with the same complex implementation and integration problems and not even notice them, and if it’s not quite right they lose a million potential worldwide users for every mistake.

The recent success of Ubuntu as Open Source Software shows that a lot of other projects still don’t get it. The first thing an OS community needs is outreach: scratching other people’s itches and not their own.

Actuallly, he says almost exactly the opposite, but then: he emails people with patches, so he’s probably a software author himself. At the very least, he’s in that four percent, preaching to the rest of the already converted. Meanwhile, the rest of the market went thataway. And if Warren Buffett were to take a break from spreading thickly his easily-believed homespun down-to-earth nonsense for a minute or two, what would he really do to capture that 96%?

Christopher Hinze has written a great Firefox addon that exports bookmarks to OPML 1.0. Unfortunately, OPML is a bit of an anything-goes specification. So although Hinze’s plugin produces valid OPML, it isn’t the same sort of valid OPML that Google Reader expects. Google Reader, in fact, gags and chokes on Hinze’s OPML, and refuses to import it.

The main problem is that the <outline/> element, the basic hierarchical building block for OPML, will take any attributes. What does that mean in practice? Well, here’s what Hinze’s export produces:

<outline text=”Coding”>
  <outline type=”link” text=”Joel on Software” url=”http://www.joelonsoftware.com/rss.xml” />
</outline>

and here’s the result of Google Reader exporting its own store of RSS bookmarks:

<outline title=”Coding” text=”Coding”>
  <outline text=”drupal.org - Community plumbing”
    title=”drupal.org - Community plumbing” type=”rss”
    xmlUrl=”http://drupal.org/node/feed” htmlUrl=”http://drupal.org”/>
</outline>

To a computer, these are fundamentally two different data formats: the URLs are stored in different attributes, and there are attributes on each that either have different values or are not present on the other. Someone did a DTD for OPML: looking at those two apparently analogous fragments above you have to ask yourself why they bothered.

Help is at hand, though. This sort of problem is bread and butter to XSLT, and here’s an XSL transform for converting Firefox OPML to Google Reader OPML. If you have xsltproc installed on your system, you would type:

xsltproc http://www.jpstacey.info/applications/google/ff2gr_opml.xsl bookmarks.opml > fixed_bookmarks.opml

Or download the XSLT—it’s released under GPL2—and run it locally, changing that URL there to a local file location.

One thing to note: the XSLT will remove an outline wrapped around your bookmarks with title “Sage Feeds” (case-sensitive). So you can export that branch of your bookmarks, and the XSLT will strip the wrapper off and you won’t import a load of bookmarks tagged “Sage Feeds”. If you don’t like this behaviour then either rename your Sage bookmark container, or learn XSLT: it won’t kill you.

There’s a brief README.txt in the zipped archive linked above, but the theme’s main features are:

• XHTML compatible (in core theme files)
• Minimal, semantic markup
• No sidebar
• Excerpts on archive and category pages
• Implicit RSS feeds: the only orange icon is in your browser chrome
• Adaptive top navigation
• Separate pages for archives, categorisation and blogrolls
• Next/previous rel links in header
• Support for special pages e.g. blogroll, tag cloud

The todo list includes:

• Implement my Blogthis! plugin, while trying to keep minimalist
• Unobtrusive hiding of elements, using jQuery
• Improve styling

The theme is in a fairly alpha state. The PHP is fairly straightforward, apart from some neat theme functions, but don’t blame me if everything goes bang.

Alsa seems stable and graphics support is present from installation onwards. Enabling fancier 3D compiz effects requires the nvidia-glx-new package; compiz spots this, however and prompts for installation. All very smooth. Wireless works; my VoIP headset works; but I haven’t yet tested Bluetooth.

The only problem was in upgrading from Gutsy: my previous peregrinations had rendered my hybrid distribution shafted and incapable of upgrade. This isn’t a problem, though, if one has installed the /home directory (and in my case the /music one too) on a separate partition: the Ubuntu Live CD will blat the root partition with Heron, but leave the other partitions alone if you so require. Don’t resize any of your partitions during installation, though, or you’ll lose everything. Everything!

It’s called Straight Edge, and I’ve switched to it today. Once I’ve finished alpha-testing it I’ll write more about it, and offer it for download if anyone’s interested.

That works just fine for most purposes, but we may have to bring in content from not just outside the core Drupal tables but outside the core database, and even on a remote server through webservices. To this end we’ve decomposed the core user module’s user_load() and user_save() functions. This helps us understand better both the workflow and at what points in it our own code can motor into life, query all those extra resources (or set those queries in motion), assemble the rest of the user++ object, and then hand control back over to Drupal.

For those who don’t know much about Drupal, its core has a hook-based API structure. At certain points in its workflow, it checks all the modules for functions following certain naming conventions (typically the module name followed by the hook name e.g. mymodule_init on response startup, or mymodule_block to return details about the module’s support for Drupal page furniture). Any matching hook functions are called in the order defined by module weightings, and then page processing will generally continue: you can crowbar a grind-to-a-halt exit() in your hook, but it’d be unwise, as you can never be sure what tidying up Drupal might need to do after your hook. Outside these hooks, your code has little control over Drupal’s core functioning, unless you stub out entirely the bits of core you need yourself, and get your request to use those bits instead.z

Because of the way they let your code tag along with Drupal’s powerful core, hooks are essential to developing modules in the most Drupalish way. With that in mind, here are flow diagrams of the three basic aspects of user functionality—create new, load, save existing—lifted straight from examination of the code:

• Loading an existing user with user_load
• Creating a user: saving a new user with user_save
• Updating a user: saving an existing user with user_save

Although you have to save a user before you can load them, I’ve put this functionality first in the above (admittedly unordered) list. There are two main reasons for this:

1. user_save actually calls user_load a number of times, once or twice, to “refresh” the user object
2. user_load is a more primitive function and so bears examination first

Stripped down, user_load consists of: querying the database for a core user record matching the search criteria; returning this and the extended profile data; unserializing a free-data field and inserting it into the user object; discovering user roles; triggering hook_user('load') and returning the object (or boolean false, if no user found).

What this reveals (which I didn’t realise before) is that the anonymous user is in the Drupal users table, with ID=0. Otherwise, searching for this user would return no records, and the anonymous user object could not be instantiated. You could therefore attach rich data to the anonymous user, if you were in a hacky mood.

The two user_save workflows are fairly similar. Creating a user means obtaining an ID from the database: because some MySQL providers have poorer feature sets than others, referential integrity is ensured at the application level rather than the database level. In place of obtaining an ID, user update calls hook_user('update') to pre-process the user. Both workflows then set aside special fields, such as the user’s password, user roles and any profile fields managed by that module (determined from user_fields()). Then they save this data into the database in slightly different orders, with user creation calling hook_user('insert') early on, and the update procedure calling hook_user('after_update') much later in the process, just before determining the external authentication mappings (e.g. OpenID) and returning the user object.

What does this mean for us? Well, we’ll want varying amounts of data to piggyback on the core user object, so we have somewhere to cache it. Ideally this data won’t be summoned—brought out of the distributed data ‘cloud’—on every request/response cycle, so we’ll need to do some local cacheing, but not so much that we’ll get out of synch with the cloud (or that we’ll duplicate sensitive data). We think that, given the pair of hooks in user_save for existing users, we’ll have just enough leverage to do this: the first hook will effectively “tear down” our extra data, so we can do what we want with it, and store it somewhere temporarily; the second hook will “set up” the user for the rest of the request, by putting all that data back in. The existence of user_load within user_save complicates things somewhat, but at the same time it gives us some more wiggle room, because each call to that function fires another hook.

A Drupal hook is worth a thousand lines of module code, but they’re still a bit few and far between for some workflows. Hopefully the accompanying images will help anyone reading to find them, and ditch those thousand lines before they’re even written.

First, a bit of background. Postcode Anywhere provides a per-lookup paid webservice, converting UK postcodes into valid house names/numbers, streets, towns etc. It allows you to accept a user’s submission of a postcode to e.g. a signup form, then present them with a list of sample addresses, rather than them having to fill in all of their address details.

There’s only really one method exposed for the PA functionality, and it takes a text postcode and returns an array of results:

$data = pamb_pa_get_bypostcode($postcode);

MailBuild and CreateSend, on the other hand, provide a way to store email lists and send newsletter campaigns to subscribers. However, our client also wanted to use it to store snail-mail addresses, so they could use it for both requests for printed matter and also to keep subscribers up to date via email. It was possible to implement this using custom fields, of which the service permits around a dozen: address line 1, address line 2 etc.

There’s an existing Mailbuild module, which technically does the trick, but it’s overkill compared to what the client wanted. pamb’s integration is essentially just “push”: it passes the results of a form submission to CreateSend, by using a template (the one you’ve moved to your theme directory) to “theme” the data as an XML packet. If there are no problems then the system returns nothing; otherwise, it returns a structure of data including the HTTP response and the SOAP packet from the request.

Here’s some sample code:

/* Send SOAP packet */
$no_probs = pamb_mb_send_soap($form_id, $form_values);

/* If OK, set a generic message and redirect: the redirect will pick up
on the generic message and know all has gone well */
if(!isset($no_probs)) {
  drupal_set_message(t(AUTO_SUCCESS));
}

/* If not OK, $no_probs contains the diagnostics */
else {
  /* do something else */
  drupal_set_message(t(AUTO_ERROR));
}

For now, for want of a better name, the module is called “pamb.” And [edit] the code is now available on Sourceforge under the GPL: that should make it compatible with Drupal’s core licence.

1. Download pamb: currently only available in subversion, but clients exist for Linux, Windows and OSX.
2. Unpack it into your modules directory
3. Edit the defines at the top of the file for your own Postcode Anywhere or MailBuild/CreateSend accounts.
4. Stick the pamb_soap.tpl.php in your theme directory
5. Switch on the module in the admin interface

You’re now ready to start developing with its handful of functions. Feedback more than welcome, as given Drupal’s minimal support for mailing-list functionality in core it’d be good to make this into a fully-functional module for similar third-party contact services, and even port it to 6.x.

[edit 2008-06-17: now hosted on Sourceforge]