django

I've never been completely happy with this spindly and slightly confusing diagram from the Django Book, ever since it appeared the first edition. Once I'd digested it, I almost immediately started redrafting it as an exercise in explaining it to others, for a possible seminar for wannabe Djangolians at Torchbox.

Time went by, as it was wont to do, and I still had a slightly incomplete diagram of the Django internal architecture on my desktop. The Django Book had since been reorganized, empires had risen and fallen, we had probably passed peak oil, and I still hadn't any use for that fecking diagram.

In an attempt to just get rid of it I've polished it up and posted it here: a three-page PDF of Django's architecture. It's got callout boxes and different colours and everything. Feast your eyes on it; move rapidly between pages, as in a flick-book; complain about the fact that it's not much of an improvement on the original. Just don't leave it on my desktop, please.

Google are introducing paid-for extensions to Google App Engine quotas, which is great as it lets you build more complex applications if you're willing to pay the rates. At the same time they're reducing the baseline free quotas. That's a shame, but only to be expected in a recession: at least there's still wiggle room there for the casual developer to play with the service.

As far as I can see there's still no SLA in the Terms and Conditions (11.3 seems to be fairly clear on this), which is a thorny issue, not just for our clients. There's also no option I can see to increase your file number quota (as opposed to increasing the quota for total disk space) which means that deploying a Django app of any complexity presumably remains a pain in the arse.

A summary of OGN9, originally on the Google Open Source Blog:

On Wednesday 22 October, over a hundred geeks attended the ninth Oxford Geek Night, upstairs at the Jericho Tavern. After the musical theme of the previous OGN, this one had a distinct flavour of online publishing.

Jeremy Ruston of BT Osmosoft demonstrated TiddlyWiki (an open-source wiki application that works offline) and revealed its offshoot Project Cecily, a prototype ZUI (Zooming User Interface). Adrian Hon of Six to Start then explained the ideas and tech behind We Tell Stories, a complex Django-based site of interactive fiction, built for publishers Penguin UK.

Continuing the Django-ish theme, Rami Chowdhury discussed WSGI—the server/application web standard—in one of the more technical microslot talks (five minutes each, from local volunteers). In another, David Sheldon took us through the steps required to hack a CurrentCost electricity meter, to get at the regular XML packets it emits from a serial port.

In the microslot sessions we also covered moving your business mail to Google Mail, protection—or otherwise—against socially engineered virus vectors, and how to use an interlocking stack of Python, Ruby on Rails and Java to crawl the web for comparisons of mobile-phone tariffs. We also had a short talk from the Oxfordshire branch of the British Computing Society about their forthcoming IT-industry events.

As usual, the evening was rounded off by a book raffle, this time courtesy of Pearson Education. Many of the night’s talks—especially the keynotes and the microslot on antivirus protection—had generated heated debate among the geeks in the room, and this carried on for some time after proceedings had officially finished.

The Oxford Geek Nights are free events, thanks to Torchbox and the Google open-source team. But even the generosity of our sponsors couldn’t prevent the upstairs bar staff from tapping their watches, as we all headed downstairs into the main room of the pub to continue arguing.

As discussed previously, at Torchbox we recently built TimeToLead.eu, an advocacy site set up by the four major environmental NGOs to prompt MEPs into passing the sort of legislation the world's climate desperately needs. The project itself needed a fast turnaround time, and its pan-European audience demanded strong i18n and l10n. This had to be in place at all layers, including an embedded Flex application.

In a parallel with the acronym LAMP, the overall stack is probably best described as "LAPD":

• Linux
• Apache
• PostgreSQL
• Django

although there are refinements at almost every level, which I'll go into below.

I18n with Django was a joy to implement. We had to have the site translated in six languages, but this was practically a doddle with Django's core internationalization behaviour. Before we'd put the translations in, I clicked on "POLSKI" and was sure the l10n wasn't working, until I spotted that the "ENGLISH" link had magically changed to "ANGIELSKI".

The only issue we've had---which we're hoping to work around---was deciding on an initial translation when the visitor arrived for the first time. Some parts of the system seemed to need l10n---the plumping for a specific version of the site---before Django's LocaleMiddleware had done that for us, so we've had to force English until the user states otherwise. I'm hoping that a version 2 rewrite will fix that.

PostgreSQL might seem an odd choice of database to the LAMP community, who are used to MySQL's minuscule overhead and often actively work around its deficiencies to keep that low overhead. But PostgreSQL's stability and maturity outweigh the performance issues which---with a little care and attention---it's possible to at least partly mitigate.

Compared to MySQL or Oracle, PostgreSQL also has the accolade of being supported by Django but having no outstanding database issues. Whether that's because nobody else uses it, or because the integration is tight and relatively bug-free, I daren't comment, but we've had nothing but seamless, transparent behaviour thus far.

Our Django-oriented hosting is with Webfaction, who are geared up for one-click mod_python Django deployments. The Django app sits in its own Apache process, while static files are served by the Linux server's main Apache process: differential serving of content lets us take advantage of differential hosting fees. The hosting is a pretty good package, although TimeToLead.eu (despite being a pretty small application) is already finding the maximum memory package a bit restrictive, so we'll need to keep an eye on that.

On top of this, Django has to a greater or lesser extent enabled us to use a whole host of other neat little technologies to integrate the site and improve both the user and the developer's experience:

• The main Flash widget is written in Flex: I'm no particular fan of Flash myself, but it fulfills the remit admirably here. Because of text flow issues, there are actually six versions of the Flash file, one for each language, but the majority of the explanatory text is then picked up from an XML feed provided by Django, so can be retranslated by the client without recompiling Flex.
• Swfobject serves up our main widget and the YouTube file. It degrades well, but bear in mind that the API to it has changed considerably in the most recent major version.
• Unit tests in Django cover a number of stress points in the code (although not all, owing to the timescale). Towards the end, we were testing any major functionality changes by writing the tests first, which I was really pleased with.
• BeautifulSoup, which I mentioned a few days ago, is employed in the unit tests to check that certain content is coming through on the front end. It also managed to help us proof against a browser-dependent FOUC we encountered during development.
• Translations are managed by the Django application Rosetta: the most recent stable version is for Django 0.96 but the bleeding-edge repository copy seems to work OK on 1.0b1

Django has been the saviour here, living up to its promise as a framework for rapid application development. No framework is ever perfect, and there'll always be arguments over the right way to proceed, reconciling what the framework developers want you to do with what the end developer wants to do, but this particular brief sojourn alongside Holovaty, Kaplan-Moss, Willison et al has been tremendous fun.

In Django, when you have a queryset---a set of objects returned by an object manager, as yet not evaluated to actual objects---and you want them ordered randomly, set your order_by parameter to be a string of a single question mark:

>>> from mysite.models import Country
>>> Country.objects.order_by('?')
>>> Country.objects.filter(continent='Europe').order_by('?')

The objects should then be iterable in a random order.

Astonishingly, the existing documentation has very little Google juice, unless you search for "randomly" rather than "random". Otherwise, at most you get db-api.txt telling you about using ordering tuples when you build your model.

Django continues to gather momentum towards its imminent 1.0 release. The 1.0 beta 1 is out; the developer documentation has been refactored; it already places nicely with Python's powerful debugging and logging tools; indeed, all is proceeding according to the roadmap, more or less. James Turnbull will be speaking about Django 1.0 at the eighth Oxford Geek Night this Wednesday, and it looks like he's got plenty of triumphs to bulletpoint for us.

An Oxford Django sprint had been mooted for this weekend. I didn't hear much more about it, but to be honest I had the great opportunity to actually have my own sprint---against 1.0b1---in work this week, working on a fast-turnaround project. I definitely felt performance improvements, especially when running unit tests. It was also lovely to work on my first internationalized/localized site and to find that it was just a question of dropping in certain bits of middleware to make it work across six languages. We didn't have any translations in place, but I clicked on "Polszczyzna" expecting bugger-all to happen and then suddenly realised that the English-language link read "Anglieski." It's characteristic of Python's (and Django's) refreshingly plastic and just-works behaviour. Magic.

We did encounter one bug, involving model inheritance. I struggled for a while with registering with the project trac to report it. It's my first mediocre experience with Django: I waited a day or so for the arrival of an account-confirmation email, but eventually gave up without adding what would have admittedly been a me-too to an existing bug report. But then, email finally in my inbox, I chased it up just now, to find that it's been fixed. Today.

Probably much like Django itself, the project's interface with the user/consumer requires some past experience with its foibles, but the actual endeavour itself is fast, well-factored and puts most closed-source equivalents to shame.

To the untrained eye, you might think that you can put any old string in as the second element of one of your Django URL dispatcher patterns in urls.py:

from django.conf.urls.defaults import *

urlpatterns = patterns(”,
    (r’^$’, ‘myapp.views.webroot’),
    (r’^a_page/$’, ‘myapp.views.another_page’),
    (r’^nothing_here_dude/$’, ‘myapp.views.this_view_does_not_exist’),
)

You have a fairly basic Django page which works; you add a new tuple to the URL patterns and refresh the page: it works OK, so if everything runs through the URL dispatcher fusebox, then it must be all right, yeah?

Unfortunately, nonexistent view methods seem to break pages that do a reverse lookup in the URL layer using the {% url %} template tag. You might not use this, but the standard Django admin interface does, several times, on its first page after login e.g:

<a href=”{% url django.contrib.admin.views.doc.doc_index %}”>{% trans ‘Documentation’ %}</a>

The above code generates a URL by comparing the view method to the evaluated second arguments of each URL pattern, and evaluation fails at the nonexistent method, giving the error:

ViewDoesNotExist at /admin/
Tried options in module myapp.views. Error was: ‘module’ object has no attribute ‘this_view_does_not_exist’

Sometimes this error isn’t thrown, so perhaps the underlying code searches a dict of URL patterns in no particular order: that would mean that every now and again it returns the matching view without ever getting to the nonexistent method.

The fix is to comment out the URL. But what if you’ve got a big codebase, and there are several people working on it, and sometimes nonexistent view methods just get committed? Testing can help here, specifically interface testing using the fake Client “browser”. That’s not easy when it comes to the out-of-box admin interface, though, because its login procedure seems to require both cookies and a hidden, hashed field to prevent automated logins.

This sounded like it was going somewhere when I started writing it; now, though, I think it’s just handy to publish it and let it have the Googlejuice, as I couldn’t find any references online to other people spotting this solution to a reasonably common problem that just—possibly because another view somewhere else was finally fixed—suddenly vanishes.

Django’s templating language is intentionally quite restrictive. The idea is that you have to do all your data munging in the control-ish sublayer of the view layer, in the method registered as handling the view in urls.py. In principle this simplifies templates, but in practice it can make life for the developer more difficult: you have to really think ahead, and assemble your variables properly, so that the templating language can use simple iterative loops to prepare your HTML.

What if you want the same output for several different data structures? Let’s say we have three people, and we’d like to run the following include, saved as name.html, on all three:

<p>{{ person.firstname }} {{ person.surname }}</p>

Ideally you’d have the three people in a structure called people, and be able to run:

{% for person in people %}{% include “name.html” %}{% endfor %}

But what if the order matters? What if you need account_director to do different things in the template from project_manager? You’d want to send them through as separate data structures, and this:

{% include “name.html” %}
{% include “name.html” %}
{% include “name.html” %}

would just do nothing, because it references person and not account_director. An alternative is to turn name.html into a custom tag, taking arguments; but if the content of name.html were any more HTML-heavy then you might want to abstract it into the include nonetheless, and then you’d just be running a custom tag method, to pass variables to a template fragment, to render the fragment, to pass it back up to the template. It’d be nicer if we could do this with core functionality instead.

Using the latest development version of Django, we can implement basic variable assignment—getting person to reference whatever we want in the template layer—using the with template tag. This means that we can wrap each include tag to produce the required output:

{% with account_director as person %}
  {% include “name.html” %}
{% endwith %}
{% with project_manager as person %}
  {% include “name.html” %}
{% endwith %}
{% with lead_developer as person %}
  {% include “name.html” %}
{% endwith %}

It seems like overkill for our name.html example, but there are plenty of situations—I just found one, hence this post—where it makes sense to factor out code into an include: being able to fiddle with the template context before including it is a lightweight alternative to template tags and a tidy alternative to dumping the HTML right there in the primary template.

I wish Simon Willison had written about Django logging and debugging earlier than today. That way I wouldn’t have used the slightly daft solution I describe in the comments.

Of course, much earlier than today, and he’d have had to have written it before giving the talk that it was based on, which might sound a bit demanding on my part. But hey! I don’t make the rules.