Your mashup is probably legal.
Your mashup is probably sound.
Your mashup is probably legal,
So pass all that data around!

Stuff here
Stuff there
And something mashed up in between (be-tween!)
Stuff here
Stuff there
And something mashed up in between

Your mashup is probably legal;
You could monetize it as well!
But though I contend it’s all legal,
Remember I-A-N-A-L!

[Repeat chorus]

That works just fine for most purposes, but we may have to bring in content from not just outside the core Drupal tables but outside the core database, and even on a remote server through webservices. To this end we’ve decomposed the core user module’s user_load() and user_save() functions. This helps us understand better both the workflow and at what points in it our own code can motor into life, query all those extra resources (or set those queries in motion), assemble the rest of the user++ object, and then hand control back over to Drupal.

For those who don’t know much about Drupal, its core has a hook-based API structure. At certain points in its workflow, it checks all the modules for functions following certain naming conventions (typically the module name followed by the hook name e.g. mymodule_init on response startup, or mymodule_block to return details about the module’s support for Drupal page furniture). Any matching hook functions are called in the order defined by module weightings, and then page processing will generally continue: you can crowbar a grind-to-a-halt exit() in your hook, but it’d be unwise, as you can never be sure what tidying up Drupal might need to do after your hook. Outside these hooks, your code has little control over Drupal’s core functioning, unless you stub out entirely the bits of core you need yourself, and get your request to use those bits instead.z

Because of the way they let your code tag along with Drupal’s powerful core, hooks are essential to developing modules in the most Drupalish way. With that in mind, here are flow diagrams of the three basic aspects of user functionality—create new, load, save existing—lifted straight from examination of the code:

• Loading an existing user with user_load
• Creating a user: saving a new user with user_save
• Updating a user: saving an existing user with user_save

Although you have to save a user before you can load them, I’ve put this functionality first in the above (admittedly unordered) list. There are two main reasons for this:

1. user_save actually calls user_load a number of times, once or twice, to “refresh” the user object
2. user_load is a more primitive function and so bears examination first

Stripped down, user_load consists of: querying the database for a core user record matching the search criteria; returning this and the extended profile data; unserializing a free-data field and inserting it into the user object; discovering user roles; triggering hook_user('load') and returning the object (or boolean false, if no user found).

What this reveals (which I didn’t realise before) is that the anonymous user is in the Drupal users table, with ID=0. Otherwise, searching for this user would return no records, and the anonymous user object could not be instantiated. You could therefore attach rich data to the anonymous user, if you were in a hacky mood.

The two user_save workflows are fairly similar. Creating a user means obtaining an ID from the database: because some MySQL providers have poorer feature sets than others, referential integrity is ensured at the application level rather than the database level. In place of obtaining an ID, user update calls hook_user('update') to pre-process the user. Both workflows then set aside special fields, such as the user’s password, user roles and any profile fields managed by that module (determined from user_fields()). Then they save this data into the database in slightly different orders, with user creation calling hook_user('insert') early on, and the update procedure calling hook_user('after_update') much later in the process, just before determining the external authentication mappings (e.g. OpenID) and returning the user object.

What does this mean for us? Well, we’ll want varying amounts of data to piggyback on the core user object, so we have somewhere to cache it. Ideally this data won’t be summoned—brought out of the distributed data ‘cloud’—on every request/response cycle, so we’ll need to do some local cacheing, but not so much that we’ll get out of synch with the cloud (or that we’ll duplicate sensitive data). We think that, given the pair of hooks in user_save for existing users, we’ll have just enough leverage to do this: the first hook will effectively “tear down” our extra data, so we can do what we want with it, and store it somewhere temporarily; the second hook will “set up” the user for the rest of the request, by putting all that data back in. The existence of user_load within user_save complicates things somewhat, but at the same time it gives us some more wiggle room, because each call to that function fires another hook.

A Drupal hook is worth a thousand lines of module code, but they’re still a bit few and far between for some workflows. Hopefully the accompanying images will help anyone reading to find them, and ditch those thousand lines before they’re even written.

from django.conf.urls.defaults import *

urlpatterns = patterns(”,
    (r’^$’, ‘myapp.views.webroot’),
    (r’^a_page/$’, ‘myapp.views.another_page’),
    (r’^nothing_here_dude/$’, ‘myapp.views.this_view_does_not_exist’),
)

You have a fairly basic Django page which works; you add a new tuple to the URL patterns and refresh the page: it works OK, so if everything runs through the URL dispatcher fusebox, then it must be all right, yeah?

Unfortunately, nonexistent view methods seem to break pages that do a reverse lookup in the URL layer using the {% url %} template tag. You might not use this, but the standard Django admin interface does, several times, on its first page after login e.g:

<a href=”{% url django.contrib.admin.views.doc.doc_index %}”>{% trans ‘Documentation’ %}</a>

The above code generates a URL by comparing the view method to the evaluated second arguments of each URL pattern, and evaluation fails at the nonexistent method, giving the error:

ViewDoesNotExist at /admin/
Tried options in module myapp.views. Error was: ‘module’ object has no attribute ‘this_view_does_not_exist’

Sometimes this error isn’t thrown, so perhaps the underlying code searches a dict of URL patterns in no particular order: that would mean that every now and again it returns the matching view without ever getting to the nonexistent method.

The fix is to comment out the URL. But what if you’ve got a big codebase, and there are several people working on it, and sometimes nonexistent view methods just get committed? Testing can help here, specifically interface testing using the fake Client “browser”. That’s not easy when it comes to the out-of-box admin interface, though, because its login procedure seems to require both cookies and a hidden, hashed field to prevent automated logins.

This sounded like it was going somewhere when I started writing it; now, though, I think it’s just handy to publish it and let it have the Googlejuice, as I couldn’t find any references online to other people spotting this solution to a reasonably common problem that just—possibly because another view somewhere else was finally fixed—suddenly vanishes.

What if you want the same output for several different data structures? Let’s say we have three people, and we’d like to run the following include, saved as name.html, on all three:

<p>{{ person.firstname }} {{ person.surname }}</p>

Ideally you’d have the three people in a structure called people, and be able to run:

{% for person in people %}{% include “name.html” %}{% endfor %}

But what if the order matters? What if you need account_director to do different things in the template from project_manager? You’d want to send them through as separate data structures, and this:

{% include “name.html” %}
{% include “name.html” %}
{% include “name.html” %}

would just do nothing, because it references person and not account_director. An alternative is to turn name.html into a custom tag, taking arguments; but if the content of name.html were any more HTML-heavy then you might want to abstract it into the include nonetheless, and then you’d just be running a custom tag method, to pass variables to a template fragment, to render the fragment, to pass it back up to the template. It’d be nicer if we could do this with core functionality instead.

Using the latest development version of Django, we can implement basic variable assignment—getting person to reference whatever we want in the template layer—using the with template tag. This means that we can wrap each include tag to produce the required output:

{% with account_director as person %}
  {% include “name.html” %}
{% endwith %}
{% with project_manager as person %}
  {% include “name.html” %}
{% endwith %}
{% with lead_developer as person %}
  {% include “name.html” %}
{% endwith %}

It seems like overkill for our name.html example, but there are plenty of situations—I just found one, hence this post—where it makes sense to factor out code into an include: being able to fiddle with the template context before including it is a lightweight alternative to template tags and a tidy alternative to dumping the HTML right there in the primary template.

Django has its own library for building forms, currently called newforms (to distinguish it from the old library, deprecated but left around for compatibility reasons). The ModelForm class in this library interrogates a particular class from the model layer and builds an HTML form—complete with post-submit validation, field widgets, the lot. The beauty of this is that you can run:

import django.newforms as forms
from myproject.models import MyObject

class MyForm(forms.ModelForm):
    class Meta:
        model = MyObject

and that’s it: you can then use an instance of MyForm anywhere in your view layer to add or edit MyObject instances in the database.

Problems arise, of course, when the automatically generated form doesn’t provide exactly what you want. It might, for example produce select dropdowns for you with thousands of items, whereas with better knowledge of the page your user is on then you might be able to whittle that list down to a handful of items.

Let’s imagine your objects can belong to Django users i.e. in your models.py you have:

class MyObject(models.Model):
    owner = models.ForeignKey(User)

This means that when you try to create an object, the HTML from newforms will give you a select dropdown of all the users in the database. But what if you have loads of inactive users, cluttering up the dropdown? How do you get rid of them with the minimal amount of coding i.e. without having to build your own form if possible?

Luckily, newforms sticks the queryset it creates somewhere that you can fiddle with it. With lazy evaluation, you get a chance to change the queryset before it hits the database; with chaining, you don’t have to worry about rebuilding the queryset yourself, but can just add to it instead.

In the MyForm class, you can override the inherited __init__ method as follows:

def __init__(self, *a, **kw):
    forms.ModelForm.__init__(self, *a, **kw)
    self.fields['owner'].queryset = self.fields['owner'].queryset.filter(is_active=True)

This calls the existing superclass constructor first, passing along any values. Then, after the queryset has been created for the select dropdown but before it’s been evaluated, the queryset is modified to whittle down the resultant set of users to only those where the “is_active” flag is set to True.

Laziness and chaining make Django’s ORM layer very effective; forms and newforms help to greatly reduce the amount of coding required by the view and control layers; together they can really simplify a codebase.

sp_rename
  [ @objname = ] ‘object_name’ ,
  [ @newname = ] ‘new_name’
  [ , [ @objtype = ] ‘object_type’ ]

So say you have a table called t_est, with a column in it called est_client. You want to rename these to t_job and job_client respectively. Firstly, with some trepidation, you rename the table:

sp_rename ‘t_est’, ‘t_job’

You find, miracle of miracles, that it works! SQL Server warns you about stored procedures breaking, but looking at other dependent tables you find that foreign key constraints have all been updated so you still have referential integrity.

At this point, you might try renaming a column called, say, est_client to job_client as follows:

sp_rename ‘est_client’, ‘job_client’

That’s wrong, of course, as you soon find out: the object_name needs to be qualified to distinguish it from . So you might try:

sp_rename ‘t_job.est_client’, ‘t_job.job_client’

Well, that was successful, but: oh! You now have a column called t_job.job_client, in the t_job table. It looks like new_name shouldn’t have been qualified after all: you should have assumed inconsistency where consistency was implied by the spec. Well, dealing with Microsoft’s tech documentation is certainly a learning experience at the best of times, so chalk this one up to experience. But if you now try to learn from your mistakes and revert the column name, you find you can’t, because:

sp_rename ‘t_job.t_job.job_client’, ‘job_client’

tries to rename the column job_client, owned by user t_job, in table t_job… to job_client. After much headscratching, you finally realise that you can employ double quotes to qualify the column name and make SQL Server treat the dot as a literal dot and not a separator:

sp_rename ‘t_job.”t_job.job_client”‘, ‘job_client’

You sit back, grudgingly satisfied. More time passes. Spring arrives. The owls are not what they seem.

Matthew Somerville is well known for his accessible takes on rubbish websites: most useful is Traintimes, a layer on top of one of the equally poor commercial British rail sites; most notorious is Accessible Odeon, a fixing of the Odeon Cinema’s website that put their own substandard development to shame so much that in 2004 they had Matthew’s version taken down with legal threats. Remember: just because you’re successful doesn’t mean you’re not stupid.

Recently, Matthew gave an OGN talk on split tickets. These are train journeys which one cross-country train company sells at exhorbitant rates, whereas the components of the journey can be bought separately from the local companies for considerably less. This is what we infer; the rail websites try to keep it all quiet, in the hope that you might lose interest and stump up. The whole headshakeworthy situation is either a stinging indictment of the stupidity of privatizing a rail network and making passengers jump through ridiculous hoops to glean even the tiniest advantage, or a perfect demonstration of how the choice-enriched consumer can leverage capitalism in action: take your pick.

To come to the point: Matthew has set up a wiki where people have been adding the split tickets they’ve worked out in an ad-hoc fashion. Last Sunday I added K’s astonishing 47% saving for Oxford–Cardiff to the end of it, and there’s been little activity since. The page hasn’t got fantastic Google juice—”split tickets” means too many other things—and is just non-dynamic HTML, editable through the wiki but otherwise searchable only by eye.

At the moment, of course, there’s no reason for Matthew to expend any more effort on such a small and barely popular data set. But last Sunday I had a sudden instinctive jolt, to the effect that: more people would be likely to take advantage of split tickets if there was an easier way of looking up other people’s discoveries (my colleagues were recently trying to find split tickets and the wiki page was harder to use than Traintimes, for that very purpose).

With this in mind, here’s my take on accessificating the wiki page: Spliticket. It accesses a cached version of the page, and then using Python’s HTMLParser to hack away at the HTML it returns either a HTML or XML representation of any journey it finds. The idea is that it’s a bit easier to use on your mobile device, and lets you pin down journeys better. I’ve also included an option for searching strictly for the same journey: I’m not sure if split tickets—even returns—always work when the journey is reversed.

Spliticket also supports friendly URLs (inspired by Traintimes itself), so York to Edinburgh in HTML becomes <http://www.jpstacey.info/applications/spliticket/html/york/edinburgh/> and the aforementioned Oxford to Cardiff route in XML becomes: <http://www.jpstacey.info/applications/spliticket/xml/oxford/cardiff/> .

Mostly this was just something to fill idle hours, and also to convince myself of some ideas I’ve had recently about loose coupling, data reuse and open data. Hopefully at the same time (a) Matthew won’t take it as a dig, (b) others might find some casual use for it see it, and (c) a select few might see it as a testament to the ease of freeing information from solid if unsemantic markup. And maybe in six months’ time we’ll all be booking split tickets as a matter of course; by then, of course, Spliticket will probably be gratefully obsolete, replaced by the fully-fledged application that poor, embittered rail passengers deserve. I’m sure Matthew will build it if so.

Last.fm’s web interface isn’t optimized for the sort of “media station” use we put it to, and their downloadable widget won’t work on the antiquated version of OSX that I’ve just about managed to squeeze onto dear old Purpurea. So today I’ve spent some time writing a bookmarklet to allow me (and you, dear reader) to hook into Last.fm and its Flash player’s behaviour, to update a larger-view popup window with the necessary information.

Here’s the bookmarklet—which with attention to catchy names I’ve called LastJS—with spaces added for legibility:

javascript:void(function(){
  var s1=document.createElement(’script’);
  var s2=document.createElement(’script’);
  s1.src=’http://www.jpstacey.info/applications/lastjs/jquery.js’;
  s2.src=’http://www.jpstacey.info/applications/lastjs/last.js’;

  var h=document.getElementsByTagName(’head’)[0];
  h.appendChild(s1);
  h.appendChild(s2);
}())

(As the above suggests, the bulk of the code is at http://www.jpstacey.info/applications/lastjs/last.js.)

And here’s what you need to do to use it:

1. Add this link to your browser favourites (IE7) or bookmarks (everyone else): LastJS bookmarklet.
2. While on a Last.FM “now playing” page e.g. for The Hold Steady, click the bookmarklet.
3. This will pull an instance of jQuery and LastJS off my own server, and add a “click for big” link under the title.
4. Click on the link. A new window will open, containing the artist information (track information is a bit obfuscated until the Flash player loads the next track).
5. Wait. As the track changes, the new window will update with the new track information.
6. The text is purposefully large yet not too large. If you’ve got a big monitor you can make it even bigger by increasing the base font size in your browser. To do this in Firefox press CTRL-+ (or Apple-+); I think it’s something similar in IE7.

Feel free to poke around at the code, and distribute it (under GPL, for what it’s worth: I tried looking into precisely which variant I wanted but my knees started to ache). Here are some of the problems it has to solve:

1. Last.fm uses Prototype, and so needs jQuery to restore the $ variable using jQuery.noConflict().
2. Events in the Flash player trigger functions in the window scope: by putting a new method in for a given event handler you can hook into Flash events. Care has to be taken, though, not to wrap and re-wrap too often—the bookmarklet might after all be re-clicked—or you might provoke a recursion warning.
3. Popup blockers mean that you can’t always guarantee the window will exist, and the code has to check accordingly.
4. Firefox seems to blank the contents of the popup window shortly after it’s opened, so any writing of text to it has to be delayed, lest it be lost by this clear-out.

All probably needs work, but feel free to have a play. It doesn’t circumvent any DRM or anything, of course, but I’ve no idea whether it invalidates terms of use, so caveat fructor.

Say you have a codebase that only you’ve worked on, and the log messages are (to say the least) idiosyncratic. Worse, they might contain swear words like “p*mh*le,” “cl*ff-pr*nker” or “Gr**nsp*n.” How can you, for example, get rid of a bit of bitching between programmers like:

$ svn log -r 200
————————————————————————
r200 | jps | 2007-05-30 14:50:56 +0100 (Wed, 30 May 2007) | 2 lines

oh for heaven’s sake - cacheing at view layer broke CSS!
Which buffoon wrote l51-78, ***Malcolm***?
————————————————————————

The svn propset command in principle lets you change log messages. However, most archives are set up to prevent this by default with “hooks.” You can alter the configuration of the archive to permit this, if you’ve got administrator’s access to the physical files of the archive. But if you’ve got that, then you can change the log message directly with the svnadmin setlog command and the --bypass-hooks option:

% echo “Fixed” > /tmp/newlog.txt
% svnadmin setlog --bypass-hooks /home/svn/repositories/bluefish /tmp/newlog.txt -r 200

The advantage of this is that all hooks remain in place afterwards, so you don’t have to worry about re-securing the archive. As the link to the Redbean book up there states, though, be warned that --bypass-hooks bypasses all hooks: if you have email alerts set up to report each submission, then they too will fail to fire. Another reason not to do it permanently, I suppose.

(And I don’t work with anyone called Malcolm.)

… This brings us to the second obviously-bad thing that can go wrong with code bases: copy and paste. It doesn’t take very long for programmers to learn this lesson the hard way. It’s not so much a rule you have to memorize as a scar you’re going to get whether you like it or not. Computers make copy-and-paste really easy, so every programmer falls into the trap once in a while. The lesson you eventually learn is that code always changes, always always always, and as soon as you have to change the same thing in N places, where N is more than 1, you’ll have earned your scar….

Code reuse is great, although some languages punish you for it: Coldfusion takes a performance hit on behalf of debug logging whenever it crosses a component boundary, even if debugging is turned off. And a lot of includes scattered across the place can make it difficult to track what’s going on (unless you use those very IDEs that Steve suggests treat code like heaps of dirt to be rearranged). I suppose ultimately it’s about developing an instinct for balancing short-term performance, ongoing maintainability and long-term scalability, and you only learn that by dint of many, many scars.