But what makes it all far, far worse is that IE6 and IE7 can sometimes produce error messages which, as is usual for those browsers, contain no useful information whatsoever.

Here’s a recent example:

Line: 64432871
Char: 2
Error: Expected identifier, string or number

See that impossibly high line number? That’s a result of the parser being struck soundly on the head by the syntax error, and consequently dribbling its way past the end of the Javascript file. IE’s incomprehensible “English” error message conspires with circumstance to make the whole report of no help at all.

Something like JSLint, on the other hand, is of tremendous help, and we’ll be running as much Javascript through it as possible in future. JSLint is just as unforgiving, but as it’s reliably unforgiving and incredibly communicative with it then that’s a bonus rather than a detriment.

It’s sometimes a little too strict to be useful, complaining about implicit global variables (even when that variable is called window). My suggestion is to ignore those reports, though; but watch those commas!

That works just fine for most purposes, but we may have to bring in content from not just outside the core Drupal tables but outside the core database, and even on a remote server through webservices. To this end we’ve decomposed the core user module’s user_load() and user_save() functions. This helps us understand better both the workflow and at what points in it our own code can motor into life, query all those extra resources (or set those queries in motion), assemble the rest of the user++ object, and then hand control back over to Drupal.

For those who don’t know much about Drupal, its core has a hook-based API structure. At certain points in its workflow, it checks all the modules for functions following certain naming conventions (typically the module name followed by the hook name e.g. mymodule_init on response startup, or mymodule_block to return details about the module’s support for Drupal page furniture). Any matching hook functions are called in the order defined by module weightings, and then page processing will generally continue: you can crowbar a grind-to-a-halt exit() in your hook, but it’d be unwise, as you can never be sure what tidying up Drupal might need to do after your hook. Outside these hooks, your code has little control over Drupal’s core functioning, unless you stub out entirely the bits of core you need yourself, and get your request to use those bits instead.z

Because of the way they let your code tag along with Drupal’s powerful core, hooks are essential to developing modules in the most Drupalish way. With that in mind, here are flow diagrams of the three basic aspects of user functionality—create new, load, save existing—lifted straight from examination of the code:

• Loading an existing user with user_load
• Creating a user: saving a new user with user_save
• Updating a user: saving an existing user with user_save

Although you have to save a user before you can load them, I’ve put this functionality first in the above (admittedly unordered) list. There are two main reasons for this:

1. user_save actually calls user_load a number of times, once or twice, to “refresh” the user object
2. user_load is a more primitive function and so bears examination first

Stripped down, user_load consists of: querying the database for a core user record matching the search criteria; returning this and the extended profile data; unserializing a free-data field and inserting it into the user object; discovering user roles; triggering hook_user('load') and returning the object (or boolean false, if no user found).

What this reveals (which I didn’t realise before) is that the anonymous user is in the Drupal users table, with ID=0. Otherwise, searching for this user would return no records, and the anonymous user object could not be instantiated. You could therefore attach rich data to the anonymous user, if you were in a hacky mood.

The two user_save workflows are fairly similar. Creating a user means obtaining an ID from the database: because some MySQL providers have poorer feature sets than others, referential integrity is ensured at the application level rather than the database level. In place of obtaining an ID, user update calls hook_user('update') to pre-process the user. Both workflows then set aside special fields, such as the user’s password, user roles and any profile fields managed by that module (determined from user_fields()). Then they save this data into the database in slightly different orders, with user creation calling hook_user('insert') early on, and the update procedure calling hook_user('after_update') much later in the process, just before determining the external authentication mappings (e.g. OpenID) and returning the user object.

What does this mean for us? Well, we’ll want varying amounts of data to piggyback on the core user object, so we have somewhere to cache it. Ideally this data won’t be summoned—brought out of the distributed data ‘cloud’—on every request/response cycle, so we’ll need to do some local cacheing, but not so much that we’ll get out of synch with the cloud (or that we’ll duplicate sensitive data). We think that, given the pair of hooks in user_save for existing users, we’ll have just enough leverage to do this: the first hook will effectively “tear down” our extra data, so we can do what we want with it, and store it somewhere temporarily; the second hook will “set up” the user for the rest of the request, by putting all that data back in. The existence of user_load within user_save complicates things somewhat, but at the same time it gives us some more wiggle room, because each call to that function fires another hook.

A Drupal hook is worth a thousand lines of module code, but they’re still a bit few and far between for some workflows. Hopefully the accompanying images will help anyone reading to find them, and ditch those thousand lines before they’re even written.

Say you have a codebase that only you’ve worked on, and the log messages are (to say the least) idiosyncratic. Worse, they might contain swear words like “p*mh*le,” “cl*ff-pr*nker” or “Gr**nsp*n.” How can you, for example, get rid of a bit of bitching between programmers like:

$ svn log -r 200
————————————————————————
r200 | jps | 2007-05-30 14:50:56 +0100 (Wed, 30 May 2007) | 2 lines

oh for heaven’s sake - cacheing at view layer broke CSS!
Which buffoon wrote l51-78, ***Malcolm***?
————————————————————————

The svn propset command in principle lets you change log messages. However, most archives are set up to prevent this by default with “hooks.” You can alter the configuration of the archive to permit this, if you’ve got administrator’s access to the physical files of the archive. But if you’ve got that, then you can change the log message directly with the svnadmin setlog command and the --bypass-hooks option:

% echo “Fixed” > /tmp/newlog.txt
% svnadmin setlog --bypass-hooks /home/svn/repositories/bluefish /tmp/newlog.txt -r 200

The advantage of this is that all hooks remain in place afterwards, so you don’t have to worry about re-securing the archive. As the link to the Redbean book up there states, though, be warned that --bypass-hooks bypasses all hooks: if you have email alerts set up to report each submission, then they too will fail to fire. Another reason not to do it permanently, I suppose.

(And I don’t work with anyone called Malcolm.)

… This brings us to the second obviously-bad thing that can go wrong with code bases: copy and paste. It doesn’t take very long for programmers to learn this lesson the hard way. It’s not so much a rule you have to memorize as a scar you’re going to get whether you like it or not. Computers make copy-and-paste really easy, so every programmer falls into the trap once in a while. The lesson you eventually learn is that code always changes, always always always, and as soon as you have to change the same thing in N places, where N is more than 1, you’ll have earned your scar….

Code reuse is great, although some languages punish you for it: Coldfusion takes a performance hit on behalf of debug logging whenever it crosses a component boundary, even if debugging is turned off. And a lot of includes scattered across the place can make it difficult to track what’s going on (unless you use those very IDEs that Steve suggests treat code like heaps of dirt to be rearranged). I suppose ultimately it’s about developing an instinct for balancing short-term performance, ongoing maintainability and long-term scalability, and you only learn that by dint of many, many scars.

$ r=NUM; rr=-r`expr $r - 1`:$r; svn log -r$r; svn diff $rr

It looks a bit unwieldy, but you can keep pressing the up arrow and home key, and re-editing NUM, multiple times to look at multiple changes:

$ r=320; rr=-r`expr $r - 1`:$r; svn log -r$r; svn diff $rr
. . .
$ r=321; rr=-r`expr $r - 1`:$r; svn log -r$r; svn diff $rr
. . .
$ r=322; rr=-r`expr $r - 1`:$r; svn log -r$r; svn diff $rr

Also, you can automate your checking to loop gradually over a range of changes:

$ for ((r=310;r< =336;r+=1))
> do rr=-r`expr $r - 1`:$r; svn log -r$r; svn diff $rr
> cat -
> done

Press CTRL-D whenever you’re finished looking at a given change, and the next change will come up.

Until very recently all seemed to be going well with these two jobs, until I suddenly found a rash of OpenSSH popups appearing as Ubuntu struggled in the face of them to start. Often enough of them would appear that GNOME would panic and warn me that my keyboard might have been taken over by, I don’t know, aliens, or lolcats.

It turned out that my machine was presenting itself as IPv6-addressed to one server, on which the authorised key was only set up with the domain name and IPv4 expected values. This was quick to fix: I added the IPv6 address to the entry in authorized_keys and hey! presto, the automated tasks are now running with nary a whisper.

What’s more worrying is that, from the looks of my mail backups, they haven’t run since March. Presumably it’s only been a recent update to ssh on Ubuntu that has directed these password requests to the screen: until now they were probably being lost to /dev/null . So a change that I initially thought was a pain and had reconfigured what ssh was declaring was my machine’s IP address, could actually just have been a tightening-up of the error reporting; in the end I was very glad ssh had annoyed me so much, as the annoyance has potentially saved my email backups.

Drupal is certainly very interesting. Its notion of presentation is remarkable in that, at a certain level, all content consists of homogeneous nodes, whether that consists of uploaded files, images, blog posts, taxonomy categories, or embedded YouTube videos. In addition its API for templating, both as a library of functions and as a workflow that one can hook into, probably rivals WordPress in its scope and power. At the same time, though, the implicit homogeneity makes it hard to structure fundamentally heterogeneous sites; and the API hooks are very difficult to unravel: frequently you’ll want to get at a function some ten levels deep, and probably three of those levels can be overridden by your own code, but which, and how?

I want to mention more at a later date, to do Drupal justice, but suffice it to say for now that the complex hierarchy of the hook-in workflow is almost entirely opaque in PHP, a language that provides rather terse error reporting, without the function debug_print_backtrace(). Well worth a look if you’re debugging spaghetti-code, especially when all you can see is the White Screen of Death. Sprinkle it around as the gentle British rain from heaven: lightly, but often.

Very odd, I thought. Odder still that the higher-volume command was being permitted to write much bigger files before it stopped. So… it can’t be a limit imposed by Python on files creating runaway outputs. Unless it’s a bandwidth limit, so the second command was having its input stream throttled…. No documentation, though, and very few reports of the error on the web. So what could it be?

The external commands are opened with the subprocess.Popen method in Python, which lets you specify destinations for the three standard I/O streams: input, output and error. I tried setting the latter two to None. All the output from the child processes was splatted onto the screen, and lo! the Python script ran to its natural conclusion.

It turned out that both commands were producing an on-screen counter or ticker to denote progress, and as this was writing to standard output, the buffers provided by Python to collect output and errors were filling up. Once a command is told by the shell that it can no longer write into its output buffer then it can end up stalling indefinitely until the buffer is cleared!

When I added command-line parameters to turn the ticker off, and piped the output into files rather than into any temporary storage, the whole system ran very nicely indeed. I hope to publish it here soon.

Exit gracefully: it’s not always possible to run external commands in ultra-quiet mode. The diagnostics they produce might be handy, so you can’t suppress them. However, unless you have a good reason to hang onto the output within your program—post-processing, say, to extract meaningful error messages—then you should be directing them to files. Also, look out for progress counters that don’t seem to cause very much output: just because the screen isn’t scrolling past doesn’t mean that the command-line program isn’t generating reams and reams of diagnostics onto the standard output and error streams. And if you direct them into a file then you could end up with a log of gibberish. Try to think in a single dimension and a single direction, like a stream of text: there’s no support for such cute rewind-and-rewrite diagnostics in logfiles.

Java nulls in Coldfusion

The first problem is the Great Coldfusion Error, which is that Coldfusion has no concept of null. At least, it can’t differentiate natively between null and an empty string. So while you can’t explicitly tell if a Java method has returned null, and indeed there’s very little warning of a null appearing in your program’s collection of function pointers, the second line in the below will explode if there’s no associated helper for this object:

<cfset java_object.getAssociatedHelper()>
<cfset java_object.getAssociatedHelper().callHelperMethod()>

If this threw a sensible error in a unit-testing environment I’d have solved the problem quickly, but more of that later. The way round this is to work with Coldfusion’s prediliction for (a) converting nulls to “” and (b) doing all sorts of type-casting on Java objects to integrate Java and CF at the scripting level. Therefore, a check of Len(java_object.getAssociatedHelper()) before you proceed does what you might expect for nulls, yet is also happy when the Java method returns a non-null object.

Errors from errors reporting on errors

The second major problem is that of Coldfusion’s introspection. Coldfusion attempts to latch its metadata introspection methods onto Java objects—including thrown Java exceptions—wherever it thinks is possible. Sadly, that’s far more times than it actually is possible to provide introspection. This is made worse in a unit-testing environment, where introspection is used for reporting purposes. I’m using CFCUnit, which is fairly thorough, but would frequently fail with a generic error that I couldn’t trace at all:

coldfusion.runtime.java.MethodSelectionException: The selected method getDetail was not found.
Stack trace: all from files within CFCUnit

I eventually worked out that Java’s XSLT engine wasn’t happy with an instance of the XSLT/XPath function concat() having only one argument; an error, moreover, that xsltproc failed to catch. The detail isn’t important, though, because you could see a getDetail error anywhere.

When you get this error, don’t bother with cfdump. That tag itself attempts introspection and throws the same generic error. Instead, use cftry/cfcatch to trap the error, and then dump that and quickly abort:

<cftry>
    cfset java_object.thisThrowsAGetDetailException()>
    <cfcatch>
        cfdump var=”#CFCATCH#”><cfabort>
    </cfcatch>
</cftry>

In a struct of structs of structs, of exceptions within exceptions within exceptions, I found it: CFCATCH.Cause.Exception.Exception.LocationAsString carried the precise location of the XSLT error in the file. Easy, when you know where to look.

Exit gracefully: when Coldfusion calls up the underlying Java, often you don’t have all of the usual Coldfusion debugging toolkit at your disposal. This means that any introspection of obscure Java exceptions to see what’s going on will often lead to errors itself, masking the original problem. Let the system throw the error, because the thrown error will eventually bubble up as a Coldfusion struct, with everything else on the way reduced to “safe” Coldfusion data-types, suitable for introspection by cfdump. Although Coldfusion is built on Java, its knowledge of its own foundations is not innate: someone has had to build the CF/Java interface. Don’t expect it to be any stricter or more OO-robust than the rest of Coldfusion.