From Drush make to Composer, one module at a time

Drush make is great code, and provides a really innovative way to build your codebases. Its many advantages and wide adoption are among the reasons why I use it to power Drush instance. However, the file format and way of working isn't particularly standard outside the Drupal community, making it something of a "Drupalism". Such Drupal-specific standards can tend to alienate non-Drupal developers, who land in the Drupal project(s) and think to themselves: "what the hell is this? and this?"

If possible, we should always root out Drupalisms: though they bind our community together with our shared vocabulary, they can exclude as much as they include, if not more. This spirit is behind the plan to replace core D8 functionality with Symfony where appropriate: Symfony is itself a project built on as many public standards or conventions as possible. Drupalisms made pragmatic sense, when no other PHP-wide or development-wide standard existed, but now we have so many more public resources to choose from, that reinventing things ourselves seems perverse or naive.

With that in mind, we need to be thinking sooner rather than later about the feasibility of replacing Drush make with Composer.

What Drush make gives us

One of the big advantages of Drush make - permit me some nostalgia for a moment - is that, being a Drupalism, it's very Drupal-savvy. That means that it understands what's meant by a Drupal project, and can contact drupal.org to find out more information about that project. So much of what makes your site can become implicit, and you can specify as little as you want. For example, say you want to build a codebase with just Views and Ctools - but you don't care what versions you use - then you can write a mysite.make makefile as follows:

api = 2
core = 7.x
project[] = views
project[] = ctools

Extending this pattern to include other projects would as you can see be very simple. Only if your requirements are more complex - you want to pin a module to a specific version, or use a git repository as a source - does the makefile itself get more complex.

How might Composer give us the same

The equivalent in Composer is remarkably less terse. The first point is that we need to tell Composer about the difference between modules, themes etc. so they get installed in the right subfolder. We therefore immediately need the excellent composer/installers project as one of our composer.json requirements, and each package entry needs a "type" field: "drupal-module", "drupal-theme" etc.

The second is that, while Composer has some autodiscovery too, and will check in the first instance for a Packagist repository: drupal.org is not Packagist, and does not currently provide an official packages.json endpoint, although there are a couple of unofficial alternatives that are works in progress.

This situation is understandable, because so few people use Composer for Drupal yet; but it yields a bootstrapping problem meaning that, for now, our equivalent composer.json has to be so much more verbose, if it's going to be robust and not rely on beta services:

{
  "repositories": {
    "views": {
      "type": "package",
      "package": {
        "name": "drupal/views",
        "version": "7.3.8",
        "type": "drupal-module",
        "dist": {
          "url": "http://ftp.drupal.org/files/projects/views-7.x-3.8.zip",
          "type": "zip"
        }
      }
    },
    "ctools": {
      "type": "package",
      "package": {
        "name": "drupal/ctools",
        "version": "7.1.4",
        "type": "drupal-module",
        "dist": {
          "url": "http://ftp.drupal.org/files/projects/views-7.x-1.4.zip",
          "type": "zip"
        }
      }
    }
  },
  "require": {
    "composer/installers": "~1.0",
    "drupal/views": "7.3.8",
    "drupal/ctools": "7.1.4"
  }
}

But! if we were able to reliably use one of the unofficial services, then here's what the above reduces to:

{
  "repositories": {
    "drupal": {
      "type": "composer",
      "url": "http://static.drupal-packagist.org/v0.1.0/"
    }
  },
  "require": {
    "composer/installers": "~1.0",
    "drupal/views": "7.3.8",
    "drupal/ctools": "7.1.4"
  }
}

We still need installers, to ensure the right subdirectories: but the result is much more terse, and as you can see extending the "require" JSON for new projects should be trivial.

Sounds good? Well, yes, but at the time of writing, at least one third-party service has a few issues: the major one is that it can be quite slow to resolve, even before download noticeably begins. This is possibly because these services are also handling (non-essential) dependencies: I was given a download of advanced_help and panels along with the above. The second issue is that if I include the mothership theme, it installs it in the vendor/drush/ subfolder, not the vendor/themes/ folder. And these bugs can be filed on the github repository (indeed I've recently done so!) but obviously this is less favourable than a community-supported service, where maintenance and downtime could be given more priority (and carried out by a wider team.)

Where do we go from here?

Composer is ultimately the future of automated dependency building within PHP environments, and it makes sense, if possible, for Drupal to move to using it. Right now, makefile technology is at the heart of any automated Drupal site building, for reasons which are compelling enough: these reasons would need to be trivially satisfied by Composer, for any (Drupal) developer coming to it with very limited knowledge of how to wrangle complex Composer configuration.

Drupal relies on makefiles and there's a lot of "technical debt" arising from them: converting every d.o project's makefile to use Composer is no trivial task. There needs to be automated conversion and a simple upgrade path, made available to every module maintainer and anyone who's committed themselves to using makefiles in their own environment.

Drupal.org needs its own packagist service, maintained by the community, or to automatically add d.o packages to packagist.org somehow. It needs at any rate to be a trivial process to add new Drupal projects to a JSON file: as simple as adding new lines to a makefile.

By adopting Composer - at least as a parallel technology to Drush make in the first instance - Drupal.org will give the right message to the wider Drupal community: it's OK; it's safe; you can use Composer; the more active community members will watch your back if you do. And then, with that encouragement - probably - they will.

Comments

Great right up. Having bumped up against issues using drush make I'm more than ready to switch. So far I can replicate in Composer most of what drush does with the exception of downloading specific commit versions or anything from the repo itself (as apposed to the tar ball)

Have you been able to get this working for D7?

Hi Andrew,

Thanks for the comment. Accessing a git repository a straightforward extension to the "package repository" functionality in composer.json, and that's nothing specific to Drupal. You swap out the dist contents and use source instead: I think you can specify type:vcs and it will work it out.

You can see some examples here (same link as above): just omit the dist entirely and it should work. reference is anything commit-ish: so branch, tag, commit hash. And the git URL you use is provided on the Drupal project's "Version control" tab e.g. here it is for Views.

This is how drush m2c represents anything git-ish in the makefile, so I'm pretty sure it works fine.

J-P

This was my suspicion. As I am also using composer/installers, I was hoping to piggy-back the useful feature to change the destination of the packages and write the modules into a more Drupal-like destination, such as sites/all/modules/contrib/ - instead I will just copy these "rogue" modules into the appropriate directory using post-update-cmd.

Is this the approach you have been taking as well or is there a better way?

I have converted an existing drush make project into Composer and will be testing it out in an upcoming project, using some of your very useful Composer plugins. Thank you for your work in this area, I really see it as being a more linear way to develop and build Drupal projects.

As far as I'm aware, the fact that a composer package is from a git repository doesn't preclude adding a composer/installers type, so you should still be able to do that. As I say, I'm pretty certain I wrote drush m2c to support this, so if you try running your makefile through that then it should do all that for you. If it doesn't then do raise an issue on its github project.

Also, if you use Drush Instance to run all this for you, it builds sites/all/modules and sites/default for you post-make/composer, so you don't need to embed further post-update-cmd in composer.json. It will also rebuild your codebase (its equivalent of an update is a full rebuild, but mostly from caches), preserving your uploads directory, sites/default and any other custom file locations.

Add new comment