At the time I built the HTML parser using core SAX-based HTML parsing, and it was horrible. SAX works in a basic sense, but you have to build your own internal state engine, track which elements have gone past while working out what to do with the current context, and even write rules for what to do when the underlying dumb parser encounters HTML entities: no mean feat when the document is peppered with – en dashes.

Not only was writing the rules initially a pain in the rear, but adding new rules and bugfixing the existing ones was even worse. But I lived with SAX, because I was deploying on shared hosting: I presumed that this was the best option available if I couldn’t install any new shared libraries.

Not true! I’ve just rebuilt the entire parsing layer with Beautiful Soup, a Python HTML/XML parser library which (a) is available as a single file and (b) works out a decent HTML DOM tree from pretty much anything you throw at it.

Try it yourself, if you have to do any HTML parsing.It’s astonishing; beautiful, in fact. I will never write another SAX parser ever again, which I’m sure I’ve said before.

Christopher Hinze has written a great Firefox addon that exports bookmarks to OPML 1.0. Unfortunately, OPML is a bit of an anything-goes specification. So although Hinze’s plugin produces valid OPML, it isn’t the same sort of valid OPML that Google Reader expects. Google Reader, in fact, gags and chokes on Hinze’s OPML, and refuses to import it.

The main problem is that the <outline/> element, the basic hierarchical building block for OPML, will take any attributes. What does that mean in practice? Well, here’s what Hinze’s export produces:

<outline text=”Coding”>
  <outline type=”link” text=”Joel on Software” url=”http://www.joelonsoftware.com/rss.xml” />
</outline>

and here’s the result of Google Reader exporting its own store of RSS bookmarks:

<outline title=”Coding” text=”Coding”>
  <outline text=”drupal.org - Community plumbing”
    title=”drupal.org - Community plumbing” type=”rss”
    xmlUrl=”http://drupal.org/node/feed” htmlUrl=”http://drupal.org”/>
</outline>

To a computer, these are fundamentally two different data formats: the URLs are stored in different attributes, and there are attributes on each that either have different values or are not present on the other. Someone did a DTD for OPML: looking at those two apparently analogous fragments above you have to ask yourself why they bothered.

Help is at hand, though. This sort of problem is bread and butter to XSLT, and here’s an XSL transform for converting Firefox OPML to Google Reader OPML. If you have xsltproc installed on your system, you would type:

xsltproc http://www.jpstacey.info/applications/google/ff2gr_opml.xsl bookmarks.opml > fixed_bookmarks.opml

Or download the XSLT—it’s released under GPL2—and run it locally, changing that URL there to a local file location.

One thing to note: the XSLT will remove an outline wrapped around your bookmarks with title “Sage Feeds” (case-sensitive). So you can export that branch of your bookmarks, and the XSLT will strip the wrapper off and you won’t import a load of bookmarks tagged “Sage Feeds”. If you don’t like this behaviour then either rename your Sage bookmark container, or learn XSLT: it won’t kill you.

This is wonderful news, especially because the collection is being released through a slightly adapted version of Flickr’s existing website. This means, apart from it being an established interface that millions of people already know vaguely how to use, that you can do all the Flickry things with the photos—dedicated Flickr-heads will hopefully give a more qualified response in due course—and that third-party tools should already be set up to work with the content. The meta information storage won’t particularly excite any Dublin-Core enthusiasts—a block of unstructured HTML in the standard Flickr notes field, plus of course Flickr tagging—but the whole project is still a fascinating experiment, and interesting for even the casual observer of American history. How exciting does it get? More exciting than the World of Mirth Shows?

Thinking offline for a moment, this hopefully presages more leaps forward in MLA culture. One of the first would be to remove the “NO PHOTOGRAPHS” signs from all museums. At the very least such signs could be more honest, and instead read “NO PHOTOGRAPHS; unless our security guards don’t catch you at it, in which case we’ll be blissful in our ignorance. Anyway, in five years time it’ll all be online so we don’t know why we’re bothering, to be honest….” On reflection, I suppose they would need bigger signs.

It worked very well with practically no configuration or edits, but there were a few issues with the out-of-the-box behaviour of the import script:

1. Unicode character entities were being escaped in titles, leading to the exposure of the alphanumeric code e.g. “Z&amp;#252;rich” instead of “Zürich”.
2. Whitespace in post bodies is converted to hard newlines by WordPress, and so must be excised to avoid tags being broken e.g. ‘<a [newline] href=”…”>’ becoming ‘<a <br/> href=”…”>’.
3. Multiple hierarchical categories are not supported (a known problem).
4. Although categories are created and posts are linked to them, the number of posts that a category is used in is not incremented and hence the list of categories on the front-end has zero posts for each category(possibly owing to a change between WordPress versions of how this has been handled).

I’ve come up with a number of fixes that I’ve mentioned both to Davis and on the WordPress support forums. As they’ve been greeted with an eerie silence that I’ve found typical of such forums, I’ll put them up here instead.

To fix the first three problems I created rss_to_wp, a Blosxom plugin that, along with the standard interpolate_fancy package, you can use to wrap your title and category processing bits in the RSS2.0 flavour templates. Respectively, this plugin tackles the above problems by:

1. Providing an interpolate_fancy method to unescape entities
2. Normalizing any whitespace in the body of your Blosxom posts to single spaces
3. Providing an interpolate_fancy method to convert a Blosxom-style category path into a set of category tags

You’ll need to change the Davis-recommended story.rss20 template to implement the two interpolation methods. I’ve made a sample available.

The final issue was a more knotty problem, as it was a bug in the script (possibly caused by WordPress’ handling of categories changing over time). It’s easily fixed by adding a few lines to the category-handling part of import-blosxom.php as follows:

294    if (!$exists)
295    {
296        $wpdb->query("INSERT INTO $wpdb->post2cat (post_id, category_id)
297                      VALUES ($post_id, $cat_id)");
298    }
299
300    // JPS' addition - increment count if cat ID exists
301    if ($cat_id) {
302        $wpdb->query("UPDATE $wpdb->categories SET category_count = category_count + 1 WHERE cat_ID = $cat_id");
303    }
304    // End JPS' addition

Exit gracefully: exporting and then importing—transporting?—works well if the two tasks are separable. That way the integrity of the exported data can be checked in its transitory state and any bugs worked out, before it’s imported into the new system. It’s certainly worthwhile backing up the target database for the import, as this lets you preserve any quirks of your target database if you have to dump all the imported data and start again. The standard WordPress install includes a plugin for doing this, but the command-line tool mysqldump is arguably more powerful.

I wondered this as I got far enough into a PHP publishing system that I had to start thinking about adding new content through the system (rather than just jamming it into the database by hand, which is why the previous incarnation has sadly fallen into disuse). As it’s typically configured, PHP will add backslashes to anything it doesn’t trust: hence the comment “it’s a great site you’ve got here” will, when submitted by a POST request, become “it’s a great site you’ve got here”. Whether or not your server does this automatically can be checked by calling the function magic_quotes_gpc() (I realised only the other day that “gpc” stood for “GET, POST and cookies:” I probably have some catching up to do). In performing this blanket adding of slashes, PHP prevents the unwary coder from leaving his site open to both unintentional database hiccups and intentional malevolent attacks, the SQL injection attack.

All well and good, but my application is heavily object-oriented. Such objects store whatever content you give them, as well as optionally writing it to the database. If I want these objects to persist (even for the course of a single request) then any access to their internal storage must yield sensible data: those slashes have to disappear before the articles appear in an RSS feed, or on the website itself. So when the CGI environment gives slash-added content to an object, the object needs to know to both add it to the database verbatim and to produce it for viewing with the slashes removed. It can either do this by storing it in a slash-removed state or by placing a filter on its outputs.

There’s a further complication, in that content can also be written to an object by the PHP application itself: the publishing of all my unpublished articles, for example, would change the status of their accompanying objects without reference to any CGI variable. If I assumed all of this content had had its slashes escaped, then this article, for example, would lose all of its ’ text, because the object would assume they’d been added by PHP’s internals: in my second paragraph, the “after” string would look like the “before” string, and the “before” string would instead break the database insertion. In addition, what if the server is reconfigured? Can I trust my hosting company to never change the configuration of PHP, even accidentally during an upgrade?

I found myself lost in a maze of adding, removing and then adding slashes, with no clear way of deciding. Suddenly I decided: why not use one of PHP’s major downsides—that it doesn’t support persistence of objects from one request to the next very well, and hence each action is fighting against the overhead of constantly recreating and recompiling code—to ascertain which input/output processes were the most frequent (and most public) and hence needed to be the fastest? I drew a flowchart of a typical object’s behaviour and, by identifying which channels could be safely bottlenecked, arrived at a reasonable solution to the problem.

From my phrasing it’s clear that it was a foregone conclusion: I wanted, more than anything else, for content to flow straight from the database (through the object if applicable) to the user. This content needed to stay in any object in a simple, de-slashed form, so it could flow and flow as long as the object was in existence. That meant that incoming CGI content could not be stored with its added slashes intact. Counter-intuitively, then, my solution was to undo PHP’s default safety mechanisms, unescaping the CGI content and storing it raw, and then without fail adding slashes to anything that CGI or my application wanted to add to the database. This would be my bottleneck: everything else would be as fast as it could be.

Exit gracefully: ensuring all incoming content can be added to the database safely is not necessarily the most efficient or desirable long-term solution. By examining the likely workflows for content, it’s possible to make pragmatic decisions on where content should be pre-processed and where it should be left alone. Consider all your overheads, including that of short-term programming and long-term cumulative processing time: this will vary depending on your environment. Also, if you’re aware of a safety net, over the presence of which you have minimal control, account for the possibility that someone might one day remove it.

People who don’t know how to hide rows and columns still do hide them. They just find… innovative and unexpected ways of doing so. Changing row height and column width, then protecting the cells against being resized (how they can do that and not know how to hide the cells is beyond me). Or using a filter, and then hiding the dropdowns that show that they’re filtering: it’s handy to note, as I finally discovered, that doing this results in the non-contiguous row or cell numbers turning blue.

foo
foo bar
quux
blort wuu spong

to the expanded form:

foo
foo
bar
quux
blort
wuu
spong

efficiently and cleanly, only using SQL? (In case anyone’s worried, I’ve scrubbed the data sets of any personal details they might have previously contained: any resemblance to the real Blort Wuu-Spong is entirely coincidental.)

I finally decided it wasn’t possible, and although without the pure mathematics to back me up I could have kept hunting—partial solutions involving a self-join for each whitespace splitting kept rearing their heads—what finally convinced me was comparing the behaviour of SQL with that of XSL(T). The two are more alike than you might think; and no, I don’t mean SQL and XQuery, although that easy comparison provides a clue for the underlying similarity.

In XSL(T), the XML node in your original document(s) is in a sense king: it’s considered bad form (and is at any rate inefficient) to do data management on some transient data set, created within the template. Loops work best over nodesets rather than with some sort of conditional or from/to structure. This stems from XSL(T)’s underlying functional paradigm, where each nodeset is created

Of course, it’s always possible to twist non-functional behaviour out of the stylesheet (and most real-world solutions have to take a pragmatic approach to such programmatic purity) and interpreter-specific kluges exist to node-ize strings based on some non-XML token, but the language works fastest and cleanest when it’s hanging functions off nodes.

In SQL, the equivalent to the node in an XML document is the row in a query. Rows are passed around, compared with other rows based on the content of some of their cells, tied together and discarded, but very rarely can rows be created out of thin air. The closest one gets is the LEFT/RIGHT OUTER JOIN where the ON-condition is not satisfied: then the left-hand row, rather than being discarded as in the INNER JOIN, is in a sense tied to a row of NULLs. Although that equates to it being tied to no row at all, then when the SQL99 dust settles and post-processing can begin, NULLs can be reinterpreted (Coldfusion does this without being asked, for example).

So to create new rows, one can UNION two rowsets, or entangle the rowsets with some sort of a JOIN, but in simplest, non-iterative SQL, there ought to be no easy way to make one row magically split into two, or maybe three, or maybe four, based on its textual content. It breaks the underlying principle, that rows should flow through the SQL into bit-buckets or the STDOUT tray, but shouldn’t be tossed into the stream with flamboyant verve like chillis into a stir-fry.

Exit gracefully: regardless of the data itself, the data model that a given language’s designers had in mind can have the most effect on what’s plausible to do in the language. Almost all languages evolve through proprietary extensions until they can do associative arrays, every kind of loop structure and, if left alone for long enough, GOTOs, but being able to complete a task with a given language is not the same as being able to complete it, for a sufficiently large data set, before the death of your server, your development team or the universe.

So it’s always unsettling when the client then asks for such heavily relational data to be exported to a spreadsheet.

There are many excellent reasons why applications sit on top of a database instead of a spreadsheet, but most of them are hard to explain to clients, many of whom tend to use spreadsheets only as highfalutin Word tables. I calculated that I could express the data relevant to a single provider in one spreadsheet (multiple worksheets); it would therefore take around 1000 files, which the client understandably declined to accept.

Clearly a compromise was necessary, and in this instance it came about by asking the client what they actually needed: in this case, there were two separate requirements pulling in opposite directions that, singly, were easy to solve: their technical advisor wanted a schema, or at any rate a UML diagram, so they could build a companion application along similar lines to ours; their project co-ordinator wanted a checklist of the provider data we had, to avoid duplication during import and see how to proceed with their own data audits.

Exit gracefully: After having finally teased out the two separate requirements, and dealt with one by a simple dump of the database schema, I identified the tables which could, just about, be flattened. I planned ahead by checking exactly how many were involved in each many-to-one relationship—no more than two, in the areas I agreed to flatten—and eventually was able to promise the equivalent of an address book for providers. For the task in hand, this was more than adequate.

When building a database for a client, and one which might see some reuse or multiple simultaneous use, always make sure you can justify each constraint or relation not just in terms you can understand but in ones that stress the benefit to the client’s well-tended data: there will be a benefit to the data, so this isn’t as hard as it sounds. But, as you build, look for where you might need to make quick simplifications in future. Identify the quasi-flat areas you might be able to isolate, because sooner or later the client is bound to ask for data in that shape.