framework

Summary of OxDUG, 1 May 2013

Thu, 02/05/2013 - 19:36 — jp.stacey

Drupal commerce, Drush site-upgrade and lovely internationalization extensions

Yesterday was a great OxDUG meeting as usual. Mike Harris took us on a tour round the Drupal Commerce suite of modules, and even though I've had some slight experience of it when helping out with donations work on the CLIC Sargent build that Johan led, I still found it really useful to see an alternative (and arguably more shop-like and fully featured) installation of it. I followed this with a quick presentation of drush site-upgrade, including a sort-of live demo: I merely had the upgrade command running in the background while I talked about my recent site upgrade exploits.

But the punchiest presentation was organizer Finn from Agile Collective giving a quick runthrough the internationalization additions they've made to the build of the REScoop website. Again, we've used Drupal's core i18n options (largely in D6), but really only in a two-language context; regardless, it was inspiring to see these being augmented, with the localization client, menu translations and entity translation. i18n and l10n can often be pain points in a site build, so it's really nice to find out ways of making them less problematic in future.

Tags:

Blog category:

Upgrading a blog-based Drupal site from 6 to 7

Sun, 21/04/2013 - 18:50 — jp.stacey

It wasn't straightforward, but it wasn't that difficult either.

As mentioned previously, this site was recently upgraded from Drupal 6 to Drupal 7. Here's a more in-depth post explaining how I did it, and what the experience was like. Note that I use "upgraded", as distinct from "migrated", because I didn't create a brand new D7 site and push the content into it. For better or worse, I followed the actual d.o recommended upgrade path for major releases.

To avoid having to repeat the tedious work of database upgrades and code downloads, again and again for each different permutation of preferred modules, I used the drush site-upgrade command. I can really recommend it: while it's no silver bullet, it at least permits you to set a site upgrade running, then do other more productive tasks while it completes instead of having to babysit the process. drush site-upgrade very closely follows the Drupal 7.x major upgrade procedure described in UPGRADE.txt, which means that you can at least go through its output and dissect precisely what it was doing at each point.

Below, then, is my direct experience of the upgrade process: it hardly constitutes recommended best practice, as it does detail some of my slip-ups as well. If you're going to follow something step by step, then read UPGRADE.txt instead.

Before you start: just give it a whirl

Because drush site-upgrade takes away a lot of the pain of upgrading, and because it does so in a separate D7 instance that you specify with Drush site aliases, then it's tempting to just give it a try. I did, and I can heartily recommend doing so: it's the best way of discovering the biggest, most problematic hurdles between you and a D7 site.

I think in theory drush site-upgrade should upgrade all the modules in your D6 site first, before beginning on the major site upgrade. But you should also consider doing that yourself on your current site, especially as it's a (somewhat time-consuming) no-brainer that site-upgrade is always going to have to perform.

Running this command with no options means that Drush might request your input, so you probably want to babysit this trial upgrade and see what happens.

Preparing your site(s)

Along with upgrading all the contributed modules and core in your D6 site, you should also look at a few other tasks which the initial trial upgrade should have highlighted:

Create an empty target Drupal 7 database. Unless you specify a particular configuration in your site alias for this, it will need to have the name "Site alias name with punctuation removed" + "db", and the same access credentials as your D6 website. My site alias was "@jps7.local", which meant that I needed a database called "jps7localdb".
If you have any file include()s in settings.php, remove them or comment them out. This file gets copied over verbatim into your new D7 site, so it's possible that such includes will break. Not a serious problem, but it does make the upgrade output very noisy with unnecessary PHP warnings.
Download and enable the schema module, and use it to fix any inconsistencies between the D6 database as it should be, and the database as it really is. If it looks like a load of tables have been left behind by a long-gone contrib module, download that module and explicitly uninstall it, as this will also clear up any system variables it might have set. Disable and uninstall it when you're done.
Disable your custom theme. It's not essential, but having a custom theme did lead to some loss of themeing for me on my new D7 website, even though drush site-upgrade claims to downgrade the theme to core Garland.
Disable and uninstall as many contrib modules as you can. Because my configuration was straightforward, I uninstalled the whole of Display Suite including Node Displays etc, because my trial site upgrade simply could not deal with it: there seems to be a relevant bug in the issue queue, but this was the easiest solution for me. I appreciate this was a pretty drastic move, but for a blog-based website I didn't foresee any serious consequences of redoing the DS configuration by hand. I also got uninstalled Image Gallery and CCK Redirection, although again this might be too drastic for your website. Remember to uninstall any helper modules: schema, as discussed above; but also devel, coder and any testing modules.
Delete any content types you don't use. The same goes for any input formats, taxonomies etc: anything that you really don't need to migrate. If it has potential to break your migration, get rid of it now rather than later.

Successful (or at any rate completed) site upgrade

With the above changes made to the website, I ran drush site-upgrade @site.alias --auto. This extra flag means that the process runs with as little input from you as possible: in fact, assuming you've fixed any fatal errors mentioned above

This did mean that the following assumptions were made by the Drush command:

I would fix any hacks to core (htaccess, robots.txt etc.)
I would also copy across sites/files and sites/all/libraries
The default CCK-equivalent modules in D7 were used: specifically node_ and user_reference, rather than entityreference.
Automatic CCK field migration was performed on any fields that could be found, using D7's instance of CCK and the cck_migrate module. This migration of variable reliability, but improved by assuming the defaults above.

Immediate fixes on the D7 site

Once the upgrade happened, I concentrated on just getting a working D7 site with the content looking right to the site visitor. For that, I had to do the following:

As mentioned above, my uploaded files and my libraries folder had not been copied over by the automated process, and I had to do that myself.
My themeing was largely broken and CSS-free. This might have been because I was using the core Color module. Anyway, (re-)saving Garland's theme settings fixed it straight away.
To get Geshi syntax highlighting working again, I had to enable it on the text formats, and then administer Geshi and add at least one other language to the list of available languages. I think the latter is a minor bug in the CSS workflow of Drupal's geshi module.
I also downloaded and enabled the media module and file_entity, sooner rather than later. To be honest, I couldn't tell you for certain whether this was a requirement of any of the procedures here as such. I just knew I was probably going to need something to manage documents and images.

This was the point at which I switched the codebases over on the live site, and my web presence was from that point on entirely in Drupal 7!

Improving admin and more niggles

There were still a few problems with the site; in fact, I'm still shaking a few bugs out now. None of them were really serious, but they did all have to be dealt with, and I did so as follows:

I was using the image module for image management: I know, old-skool, right? This meant that to have my images managed and appearing in Media, image nodes had to be converted to managed files. The field convert module helps with this; for a more detailed howto, read this excellent blogpost by Laura Scott.
I lost my WYSIWYG profile for full text. I recreated this manually.
Pathauto was failing because token syntax has changed radically (arguably for the better) but this was not migrated by the upgrade procedure. To fix, I manually edited the tokens to use the new syntax.
My migrated images were managed OK, but the actual page for each image entity wasn't rendering the image. This was because each image's bundle type was set to "undefined" in the file_managed table. This was probably the worst problem to solve for anyone not well versed in databases, as it required a (relatively straightforward) SQL UPDATE statement.

Embracing Drupal 7

Finally, it was time to start taking advantage of some of the simplest improvements with Drupal 7. This included the following:

Toolbar and shortcut module, to provide an "admin menu" experience across the top of the site
Contextual links, for quick links to edit e.g. blocks on a page, or a view listing.
An administrator role (under Configuration > Account settings), so that whenever a new module is enabled users with that role get all permissions for it straight away
Update emails, warning me of important security updates, once per week (although all those decisions are configurable to taste)
Media module, as mentioned above
Views had automatically upgraded to version 3, which includes a huge and improving overhaul of the UI.

There are a lot more improvements I can make going forwards, but this is where I drew a line under my personal upgrade project, and called it: success.

Summary

Upgrading a simple site from Drupal 6 to Drupal 7 is certainly comparable, if not preferable, to a content migration. The above took me around a working day and a half, split over a few nights. Personally, I can recommend it, as I felt at the end that, once everything had obviously migrated correctly, I had a very robust D7 setup.

Obviously if your site is more complicated, then automatic upgrading might be less straightforward. Almost certainly you'll need to keep Display Suite in place, which means solving the bug I encountered. And if you've got any custom code or (more likely) themeing, then upgrading this to work with Drupal 7's new APIs and themeing layer will not be at all straightforward.

But much of that would be the case whether you upgraded or merely migrated content. So do consider an upgrade: I found out from my own very meagre personal experience at the Drupalcon Copenhagen code sprint that a huge amount of work has been put into the upgrade path; it really seems to have paid off.

Tags:

Blog category:

Upgraded to Drupal 7

Thu, 18/04/2013 - 19:42 — jp.stacey

"Did you notice that?" "Notice what?"

This site was quietly upgraded to Drupal 7, in stages over the course of the past few days. Well, I say quietly: it was pretty noticeable from my perspective. But hopefully nobody else realised that it was happening, or had suddenly switched; that was the point!

Partly in order to work out whether or not it could be done, I very purposefully did an upgrade of the old D6 site, and not a migration of the content into a new D7 site. It turns out that it could indeed be done, although I'm sure the simplicity of my site has a lot to do with that.

I'll do another blogpost in a few days, about how I accomplished it, but I'm waiting for the dust to settle, the bugs to shake out etc. Speaking of which, I'm off to check that this post appears in my RSS reader of choice!

Tags:

Blog category:

You wouldn't build a spreadsheet authoring tool. So why would you build a content authoring tool?

Mon, 15/04/2013 - 13:20 — jp.stacey

There are lots of answers to that, of course; but most of them seem to assume infinite time or money.

http://petermoulding.com/drupal_versus_write_your_own_code

There are two questions that I think are easy to conflate here:

1. This is a content-based site; should I use an off-the-shelf CMS like Drupal or write my own?

2. This is not a content-based site; what should I do?

To me the answer to the first question is a no-brainer: the advent of free CMSes like Drupal and Joomla have made the concept of a CMS into commodity software. Although the ease of customization of Drupal has sometimes been overstated, I find it hard to believe people who claim that, even with customization, they couldn't get at least one off-the-shelf CMS to do what they wanted for their (often quite straightforward) site; it's worse when that's followed by the second claim that their experience meant it was a good business decision to build an entirely custom offering. In short: custom CMSes make me want to weep!

Even if you don't choose Drupal, there's very rarely a use case for writing your own CMS for a CMS-driven website: after all, what fraction of organizations would propose to write their own word-processing software to put together a spreadsheet, rather than e.g. bodging the requirements with macros?

Answering the second question is much trickier. Drupal, especially if you embed it in a well-tuned LAMP stack in preparation for high traffic, is optimized (very broadly) for delivering managed content. User interaction is best kept at a minimum for performance, as otherwise you start to find yourself jumping through quite narrow hoops (Varnish edge-side includes, anyone?) to get the site to be both performant and interactive for each and every site visitor.

Even then... as you move away from Drupal, you find yourself either having to adopt a lightweight web framework, or having to spend time rewriting even simple things like drupal_not_found(), drupal_add_js() or user_access(). Who really wants to have to wrangle HTTP headers themeslves? And yet, the more a site looks like a blended content/application offering, the harder it is to make the call. Splitting the site build up into smaller features, and building it in a way that lets you make that call on a feature-by-feature basis, is one way to forever postpone that decision; maybe having a framework like Symfony in core to call upon is another way.

Tags:

Blog category:

Pressflow minor versions and double leading slashes

Sun, 02/12/2012 - 15:00 — jp.stacey

Between Pressflow 6.22.102 and 6.22.104, there was a small but substantial change. The url() function no longer normalizes leading slashes.

This brought it back inline with Drupal 6.x behaviour, but has also led to a number of our sites showing links with two trailing slashes. Browsers interpret <a href=”//…”> as a reference to a domain, not a resource (so as if it were http://…) and this leads to broken URLs.

The problem arises when a Drupal path is passed through url() (or calling functions like l()) more than once. This can happen in your own code, or it can happen in Views: if you use Views fields, and render a path out, but then e.g. exclude it from display and use its token elsewhere.

If you’re building your own links with Views fields, don’t retrieve a node path field [path] for use, as this immediately gets rendered with a leading slash and is then unuseable as a link token elsewhere: it will get a second leading slash. Instead, get the bare node ID [nid], and build links of the form node/[nid]. When these get passed through url() one time only, they get turned into the friendly [path] aliases anyway.

Tags:

Blog category:

Read more about Pressflow minor versions and double leading slashes

Our Drupal Ladder learn sprint

Tue, 09/10/2012 - 11:46 — jp.stacey

Last weekend, ten Oxfordshire Drupalers did a learn sprint at the Torchbox offices. I’ve written a more complete discussion over on the work blog.

Drupal Ladder learn sprint: pair tuition

We had a great time: it was a gloriously sunny day, which helped, as we were able to eat lunch (sponsored by Torchbox) on the lawns next to the office. But what was best is that the day was so productive: we all ended up learning together, and by the end of it we could all contribute to Drupal on the issue queues.

Thanks, Drupal Ladder project!

Tags:

Blog category:

Read more about Our Drupal Ladder learn sprint

Migrating users and profiles with the Migrate module

Fri, 21/09/2012 - 16:51 — jp.stacey

At Torchbox we’ve been working a lot with the Migrate module recently. It’s a framework for representing relationships in the data you want to import; both relationships between bits of data, and also relationships between the data and relevant Drupal entities. It used to be a GUI-driven system, but now it seems quite code-heavy.

Which suits us fine, certainly for one-off imports which can be built most straightforwardly by developers. We had to write an import for around 4600 users, plus data which we wanted to store in three Profile2 profiles (address, subscriptions, personal information.)

Migrate did a lot of the heavy lifting for us, along with some code examples. The most useful one was this example import module, started by wusel and edited by (among others) Profile2’s maintainer joachim. As a thankyou I’ve contributed to that post a bit, tidying things up and incorporating some of the lessons we learned.

The results? A flawless migration, including mapping many CSV columns to three multi-valued entity fields for the subscriptions (basically an ORM mapping.) And it was blisteringly fast: user importing on a reasonably high-spec server was at the rate of 12592/min, and we had a peak profile import of 9954/min.

Tags:

Blog category:

Read more about Migrating users and profiles with the Migrate module

Webform integrates with Entity Token...

Wed, 22/08/2012 - 19:32 — jp.stacey

… but only from 4.x onwards, it seems.

We have a specific client requirement on Drupal 7, to use webforms to update both a third-party CRM system and local “Drupal storage”: whatever that ends up meaning.

It turns out that the excellent Profile2 module provides us with better storage than D7’s core profile module (which doesn’t even use Field API, unlike the rest of D7!) We were always expecting to have to write the CRM/local updating ourselves, but piping those profile values - Profile2 doesn’t modify the $user object in the same way as Profile - into the webforms was a chunk of work we wanted to avoid.

Luckily, as of the 4.x branch, Webform seems to support token replacement; the Entity Token module (part of the Entity API project) lets entities expose themselves as tokens; and Profile2 uses this to hook itself up to Webform. It all works pretty well, although we’re crossing our fingers for a non-alpha release on Webform’s 4.x branch

Tags:

Blog category:

Read more about Webform integrates with Entity Token...

Defining customer profile weights in Drupal Commerce

Wed, 22/08/2012 - 19:25 — jp.stacey

Drupal Commerce is a great leap forward from Ubercart. It’s especially heartening to see it use so much of other APIs - Views, Rules etc. - rather than doing its own thing. However, we think we’ve found a bug in the way that it sorts the fields and panes on the checkout page.

The commerce_customer sub-module takes customer profile types (defined via hook_commerce_customer_profile_type_info()) and prepares them for inclusion in the checkout pane. When doing so, it also transfers a weight field called checkout_pane_weight, so that in theory these types can be weighted (to make e.g. contact form appear before billing address form).

However, checkout_pane_weight is not used anywhere else in the commerce codebase as far as I can see. When commerce_checkout_panes() merges in an array of defaults, it includes a simple ‘weight’ field, but not ‘checkout_pane_weight’.

Until this bug is fixed, you can work around it with the following hook in your own module:

<?php
/** 
 * Implements hook_commerce_checkout_pane_info_alter() 
 */
function MYMODULE_commerce_checkout_pane_info_alter(&$checkout_panes) {  
  // Get weights from hook_commerce_customer_profile_type_info
  $pane_weights = cscommerce_commerce_customer_profile_type_info();
  // Loop over them and weight the assembled panes accordingly
  foreach($pane_weights as $pane_key_suffix => $weight_config) {
    $pane_key = "customer_profile_$pane_key_suffix";
    if (array_key_exists($pane_key, $checkout_panes)) {
      $checkout_panes[$pane_key]['weight'] = $weight_config['checkout_pane_weight'];
    }  
  }
}

Note that, because commerce_checkout_panes() increments pane weights by a single digit each time, you don’t get much wiggle room when you’re trying to weight your own panes and they can easily end up all at the top or all at the bottom.

So if, like me, you just want to swap round two panes, and the single-digit weight increments are messing things up, here’s an even quicker workaround:

<?php
/**
 * Implements hook_commerce_checkout_pane_info_alter()
 */
function MYMODULE_commerce_checkout_pane_info_alter(&$checkout_panes) {
  if ($checkout_panes['customer_profile_billing']['weight'] 
      > $checkout_panes['customer_profile_contact']['weight']) {
    list($checkout_panes['customer_profile_billing']['weight'],
         $checkout_panes['customer_profile_contact']['weight']) =
      array($checkout_panes['customer_profile_contact']['weight'],
          $checkout_panes['customer_profile_billing']['weight']);
  }
}

This is a simple function that swaps the two weights around in a single assignment step, and it means that if you have weights of 20,19,18,17,16… then you can seamlessly swap your two profiles around to be 20,19,17,18,16… without any recalculation.

Tags:

Blog category:

Read more about Defining customer profile weights in Drupal Commerce

Programmatically executing a 2.x View

Fri, 15/06/2012 - 10:51 — jp.stacey

TL;DR: don't do it, as such! Execute a display instead.

TL;DR: don’t do it! Execute a display instead.

Drupal Views are a good half-way house between user-friendly elegance and writing the SQL yourself. With this in mind, you might want to use them as a robust database API in your own code: you don’t need to think about precisely what’s stored in which table, but execute the view instead.

Working out precisely what you have to do to run a view isn’t straight forward. This now elderly post on Earl “Views” Miles’ blog provides some clues, but only by using page-display rendering as an example. Ideally, we want to avoid running our view through the whole theme layer, as that’s just lost CPU cycles. So what do we do to run our own view?

It turns out to be 90% easy, but 10% ever so subtle. Here is the usual stab, based on Earl’s example:

$view = views_get_view('MY_VIEW_NAME');$view->set_arguments(array($my_first_arg, $my_second_arg, ...));$view->set_display("DISPLAY_NAME");$view->execute(); // - this line is wrong.

However, that doesn’t quite work. The annoying thing is that it almost works; it just doesn’t quite work. For a start, the number of items per page remains at the default, 10. This is a clue that something else needs to be invoked. But what?

Following that clue, go to the view’s own edit page and click preview. You should see that this preview does indeed respect paging. So what gives? Try invoking debug_backtrace() in View’s set_items_per_page() method, and the devel dsm() function to dump out the contents into Drupal’s messages area.

When you click on “Preview” again to refresh it, you should find that the backtrace includes a call by $view->preview() on $view->pre_execute(), and this is what calls the method to set the paging.

The only other way to call the pre-execute function? $view->execute_display(‘DISPLAY_NAME’). It’s slightly confusing and leaves you with the worry that there might be a pre_execute_display() method that you’re also somehow omitting to call…. Also, DISPLAY_NAME ought to default to, well, “default”, but when we omitted it, paging limits were once again not respected.

Here’s the correct code:

$view = views_get_view('MY_VIEW_NAME');$view->set_arguments(array($my_first_arg, $my_second_arg, ...));$view->execute_display("DISPLAY_NAME");

Your results should be available in an array called $view->result. (note: not singular!) This should respect all paging/offset settings for your display, although of course if you actually want to retrieve page 3 of your results, you’re going to have to do a bit more programming!

Tags:

Blog category: