Workflow and tools for developing with install profiles and Drush Make

March 1, 2012

A few months ago we completed a Drupal 5 to Drupal 7 migration project for a North Carolina museum website. Actually the Drupal 5 site was more of a Frankenstein site; the previous developers had more or less built their own CMS on top of Drupal. Fortunately, the superb Migrate module made writing migration code for this project a snap.

Getting a workflow together, however, was a bit more of a challenge. We had four people working on the project: two developers, a site builder, and a themer.

Because the project was complex and contained a number of different components, we agreed that development would work best with each developer building aspects of the site on their local machine. That way my work in writing migration code would not interfere with our themer's work, nor would it bother someone working on site building.

The key ingredients to a local development first workflow are git, drush, drush_make (now included in Drush 5), installation profiles, and Features.

In this blog post, we'll review some of the workflow and tools we used for development. We'll use a fictitous "MySite" project for our example.

Workflow

Install Profile

The file mysite.profile contains the default settings for the MySite project. It defines the content types, user roles, permissions, etc that go into a default Drupal install.

Unlike Drupal 6, the Install Profile in Drupal 7 is actually a few files: mysite.profile, mysite.info, mysite.install.

Drush Make files

There are actually two drush make files: mysite.build and mysite.make .

mysite.build is brief:

1
2
3
4
5
6
7
8
core = 7.x
api = 2
projects[drupal][type] = "core"
; Our distribution
projects[mysite][type] = "profile"
projects[mysite][download][type] = "git"
projects[mysite][download][url] = "git@gitserver.com:mysite.git"
projects[mysite][download][branch] = "develop"

mysite.make contains the list of modules, themes, and libraries to download and add to the site.

Features

Features will be built to separate content from presentation. For example, if you want to export your work for "Blog functionality", one feature would contain the content type and fields, while a separate feature would contain the Views for displaying it on the site.

Git

The git repo for MySite is the basic structure for the site. The structure for the repo should not be altered, otherwise drush make won't work properly with it.

The repo looks like this:

1
2
3
4
5
6
7
8
9
10
11
/mysite (root of git repo)
/mysite/modules/contrib
/mysite/modules/custom
/mysite/themes/
/mysite.build
/mysite.make
/mysite.profile
/mysite.info
/mysite.install
/resources
/README.md
  • As you build your features, drop them into /path/to/repo/modules/custom/[name-of-your-feature]
  • Custom themes go in /path/to/repo/themes/[name-of-theme]
  • Custom modules go in /path/to/repo/modules/custom/[name-of-module]
  • Edits to the install profile go in /path/to/repo/mysite.profile
  • Enable modules in /path/to/repo/mysite.info

The Drupal 5 site is in a separate repo.

Example workflow

Let's say you want to add Webform functionality to the latest dev build. You enable the Webform module in Drupal and configure the settings to your liking.

Using Features, you then package up the relevant settings and configuration into your mysite_webform Feature and download it to /path/to/repo/modules/custom/mysite_webform.

In the mysite.info file, you would add your Feature to the list of dependencies (this will enable it on site install).

Using git, you would then add this directory and mysite.info, commit to develop, and push the code up to origin.

Then other developers can run through the site rebuild command(s) and have your Webform feature enabled and configured locally.

Tools to make development easier

Our team's comfort with using the command line or drush varies. For a local development first workflow, using the command line is essential to productive, time-efficient development. Even for those of us more comfortable with the command line and drush commands, remembering all the commands to rebuild our local environments is a challenge. So, we wrote some scripts to help with the proces.

Rebuilding the site

The main script is called rebuild.sh. By running rebuild.sh on your local machine, you can ensure that you will have (1) the latest code from the develop branch in origin, (2) your local site's database will be identical to the other developers, (3) the migration script will pull in the latest content from the source database. So, let's break it down:

The first part is a simple yes/no prompt (borrowed from the Aegir project). This makes it simple for us to ask the user if we should continue at different points throughout the rebuild process:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
prompt_yes_no() {
  while true ; do
    printf "$* [Y/n] "
    read answer
    if [ -z "$answer" ] ; then
      return 0
    fi
    case $answer in
      [Yy]|[Yy][Ee][Ss])
        return 0
        ;;
      [Nn]|[Nn][Oo])
        return 1
        ;;
      *)
        echo "Please answer yes or no"
        ;;
    esac
 done
}

The next step in rebuild.sh is to read the developers configuration from rebuild.config. This file should sit in the same directory as rebuild.sh (in our case, we kept both in the /resources directory at the root of the repository), and rebuild.config should not be tracked in the repo (a rebuild.config.example file should be though, as a helpful example to the other developers):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Reading options from rebuild.config
FILENAME=rebuild.config
while read option
do
    export $option
done < $FILENAME
DRUSH="$DRUSH_PATH"
 
# Start our rebuilding
clear
 
cat <<EOF
 
*** IMPORTANT ***
 
The following values were read from rebuild.config in your resources directory.
Please make sure they are correct before proceeding:
 
  D5_DRUPAL_ROOT = $D5_DRUPAL_ROOT
  D7_DRUPAL_ROOT = $D7_DRUPAL_ROOT
  DRUSH_PATH = $DRUSH_PATH
  D7_DATABASE = $D7_DATABASE
  D5_DATABASE = $D5_DATABASE
  D5_GIT_REPO = $D5_GIT_REPO
  D7_GIT_REPO = $D7_GIT_REPO
 
EOF
 
if ! prompt_yes_no "Are you sure you want to proceed?" ; then
    exit 1
fi

Basically, this is reading some variables that we can use later on the script. If we look at what is in rebuild.conf, we see:

[gist:1935141:rebuild.conf]

The script asks that we do indeed want to rebuild our local environment, and continues by removing the old directory, running Drush Make, and re-installing the site using our install profile.

1
2
3
4
5
6
7
8
9
10
11
echo 'Rebuilding MySite...'
echo 'Removing '$D7_DRUPAL_ROOT' directory...'
chmod a+w $D7_DRUPAL_ROOT"/sites/default"
chmod a+w $D7_DRUPAL_ROOT"/sites/default/files"
rm -rf $D7_DRUPAL_ROOT
echo 'Executing drush make'
$DRUSH make --prepare-install --force-complete $D7_GIT_REPO"/mysite.build" $D7_DRUPAL_ROOT -y
cd $D7_DRUPAL_ROOT
echo 'Re-installing site database'
$DRUSH si mysite --site-name="MySite" --db-url="mysql://root:root@localhost/$D7_DATABASE" -y
echo 'Finished rebuilding directory and re-installing site.'

On a large project (75+ contributed modules), Drush Make and a Site Install could take about 5 minutes. If you use Squid as a caching server for drush module downloads, however, the process takes only about one minute. You can also experiment with providing a "file:///" URL in your mysite.build file for the [download][url] section where you are specifying the location of the make file.

Next up, we create some symlinks from the git repo to our site build directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cat <<EOF
 
Would you like to have symlinks set up? The script will create symlinks as
follows:
  ln -s $D7_GIT_REPO/modules/custom $D7_DRUPAL_ROOT/profiles/mysite/modules/custom
  ln -s $D7_GIT_REPO/themes/mysite $D7_DRUPAL_ROOT/profiles/mysite/themes/mysitetheme
 
EOF
 
if ! prompt_yes_no 'Create symlinks?' ; then
    exit 1
fi
 
echo 'Creating symlinks'
cd $D7_DRUPAL_ROOT
rm -rf profiles/mysite/modules/custom
rm -rf profiles/mysite/themes/mysitetheme
ln -s $D7_GIT_REPO"/modules/custom" $D7_DRUPAL_ROOT"/profiles/mysite/modules/custom"
ln -s $D7_GIT_REPO"/themes/mysite" $D7_DRUPAL_ROOT"/profiles/mysite/themes/mysitetheme"
echo 'Done making symlinks.'

We continue on and migrate content, if desired. Sometimes it's helpful to rebuild the local site without migrating any content, for example, if you want to alter a field, you need to do that before any content has been added to it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# Content migration
cat <<EOF
 
The script will run 'drush migrate-import --all', which will run all
content migration patterns. You must have the Drupal 5 site setup locally
for this to work properly.
 
EOF
 
if ! prompt_yes_no "Migrate content and files?" ; then
    exit 1
fi
 
echo 'Migrating database content...'
echo 'Updating Drupal 5 git repo...'
cd $D5_GIT_REPO
git checkout master
git pull
# TODO: Set this so it imports only if git pull brought us new content
echo 'Importing Drupal 5 database...'
mysql -uroot -proot $D5_DATABASE < $D5_GIT_REPO"/database/mysite-production.sql"

As you can see, we were storing the D5 database in a git repo. Normally I would never put a database in a git repo, but because not all developers had access to the production server running the Drupal 5 site, we decided to store an ordered dump, excluding data from a number of cache tables. We wrote a script for this too; the script takes a snapshot of the D5 database, overwrites the mysite-d5-production.sql file stored in the D5 git repo, commits it to the master branch, and pushes to origin.

[gist:19035141:db_update.sh]

Ok, back to the rebuild.sh script! Next we do a couple of set up tasks for the Drupal site:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cd $D7_DRUPAL_ROOT
 
echo 'Setting date and timezone settings...'
$DRUSH vset date_first_day 1 -y
$DRUSH vset date_default_timezone 'America/New_York' -y
$DRUSH vset date_api_use_iso8601 0 -y
$DRUSH vset site_default_country 'US' -y
$DRUSH vset configurable_timezones 0 -y
$DRUSH vset user_default_timezone 0 -y
echo 'Done.'
# Run cron to make sure any initialization necessary in Drupal takes place before
# nodes are imported.
echo 'Running cron...'
$DRUSH cron

We had trouble setting some of these variables in the install profile, so we placed them in this script and had no troubles with this method.

Then, we run the migrations:

1
2
3
4
5
6
7
8
9
# Migrate isn't resolving dependencies correctly so we specify the migration order here manually
$DRUSH mi mysiteUser
$DRUSH mi mysiteBlocks
$DRUSH mi mysiteNewsNode
$DRUSH mi mysitePageNode
$DRUSH mi mysiteStaffMemberNode
$DRUSH mi mysiteFinish
 
echo 'Done.'

Then we enable our Webform feature:

1
2
3
4
5
# The Webform feature, which contains Webform nodes, is enabled after 
# all other features so as to not interfere with node IDs that are preserved
# from the Drupal 5 site
echo 'Enabling our Webforms'
$DRUSH en mysite_webforms -y

Finally we make sure the Drupal 7 site has all the files from Drupal 5:

1
2
3
4
5
6
7
8
9
10
11
12
13
# File migration
 
echo 'Copying files...'
mkdir $D7_DRUPAL_ROOT"/sites/default/files/files"
cp -Rp $D5_DRUPAL_ROOT"/files/"* $D7_DRUPAL_ROOT"/sites/default/files/files/"
echo 'Done.'
echo 'Copying images...'
mkdir $D7_DRUPAL_ROOT"/sites/default/files/images"
cp -Rp $D5_DRUPAL_ROOT"/images/"* $D7_DRUPAL_ROOT"/sites/default/files/images/"
echo 'Done.'
echo 'Finished content migration!'
 
echo 'Rebuild completed.'

End result, after running rebuild.sh I have a complete site build that is identical to the other developers, symlinks are set up, content is migrated, and I am ready to start working on features or bug fixes. When I'm done with my work, I can merge into develop and push my changes up to origin, and now every other developer can have my changes in their local environment.

Verifying make files and install profiles

One problem that cropped up a few times as we were using this workflow were commits that broke the build. For example, if someone added a module incorrectly to the mysite.make file, and added the module as a dependency in mysite.info, the build would break because Drupal would try to enable a module that actually hadn't been downloaded in the site build. So, we wrote two simple scripts, verify_install.sh and verify_makefile.sh. As their filenames suggest, the scripts allow a developer to check a makefile or an install profile before they commit to the repository.

Other tools?

So, that's the basic workflow and toolset we worked with for a larger migration project using an install profile and drush make. We'd love to hear from you if you used other tools or a different workflow.

If we did this project again, one thing I'd probably skip is the use of drush make - better just to track the entire site in a git repo. I would probably also take the time to package up these scripts as a few drush commands that made use of drush aliases, adding a line in the drush alias to specify the git repo.

Hopefully these scripts will be useful to others who are undertaking projects like this. Feel free to leave a question below!


Comments

Thanks for posting this !!!
Well... It seems that you don't actually run "drush migrate-import --all" ?
Is there a way to write down some migration dependencies/order for the migrate module to use it ?

Yes, you can specify dependencies in your migration code. For example

1
2
3
4
5
6
7
8
9
10
class MySiteProgramNodeMigration extends MySiteMigration {
  public function __construct() {
    parent::__construct();
 
    $this->description = t('Migration for Program nodes');
    // Define source fields for the migration
    $source_fields = array('nid' => t('The node ID of the Program node.'));
    // The user migration should run first, so that we can associate nodes
    // with Drupal users
    $this->dependencies = array('MySiteUser');

The Migrate Examples module has some more examples of this. We were running into issues with this though, and since we had a script in place for rebuilding the site, it was much more time efficient to call the migrations individually than to spend the time to debug our problems with Migrate module.

Ok ! I did not the migrate module well ! Thank you very much !!

Great writeup, and these scripts will be really useful, thank you.

I'm interested to hear your thoughts on keeping everything in a single git repo vs. drush make, which you mentioned in the end. Care to elaborate on that?

These scripts are awesome, and we;ve been building on them and using them a lot. Our versions are up at https://github.com/systemseed/ss_build_scripts. There are possibly a little bit more generic than the ons in this article, which were quite specific in places to the site that you were working on at the time.

Also very interested to hear some thoughts on single repo vs drush make. There are advantages and disadvantages to both approaches.

That's very cool Tom, thanks for sharing your code!

About single repo vs drush make, I've found that drush make is great if everyone on your team is up to speed and comfortable working with it. But many people are not. Having the site in a repo is conceptually simpler and you don't have to bother with symlinks and so on. Sharing changes that you've made locally is as simple as making a branch and pushing it to origin, which other developers can immediately use.

But, probably the main advantage is speed. Even using squid, waiting 1-2 minutes for a site to rebuild is too cumbersome.

Awesome write up thank you so much for sharing. Just what I need.

Hi there.

I would really like this to work.- It seems really smart.

However I can figure out the structure of the codebase. - I have a normal drupal 7 codebase ($D7_DRUPAL_ROOT) downloaded from drupal.org, and in /profiles i have made my installation profile with the same structure as you mentioned.

But when I run the ./rebuild.sh script I get to this line: rm -rf $D7_DRUPAL_ROOT , as one of the first steps. This remove anything including the script currently running!?

I dont really get this. Could you please explain what Im doing wrong.

/Steffen.

Hi Steffen,

You need to make sure that your $D7_GIT_REPO and $D7_DRUPAL_ROOT paths are in two different locations. The git repo should be in one directory, and your site build takes place in another. That way you won't end up wiping out your git repo when you run the rebuild.

You might also look into using [Drush Rebuild](https://drupal.org/project/rebuild) for implementing a rebuild workflow with Drush make.

Good luck!

Kosta

Thanks for the quick reply Kosta.

Okay but I dont really see how this is done!? I mean, when I have a drupal codebase and inside that (profiles/) have made my git repo?

When im creating multi-sites, I make symlinks in drupal's folder at /sites/ , and then have the actual site-folder located on the same 'level' as my drupal7 codebase:

Folder-structure:
drupal6 (drupal core)
drupal7 (drupal core) ( in /sites creating 'ln -s ../../ mysite')
(site)

Do you mean that I should do this as well when Im working with installation profiles?

drupal6
drupal7 ( in /profiles creating 'ln -s ../../ mysite')

Or am I on a wrong track here?

Hi,

thanks for that inspiring article. I wonder how you make a release and deploy dev to staging and prod.

Somehow you have to merge data from prod with code from dev - not sure how this could be done with install profiles..

Thanks, Michael.

I suspect the install profile is kept really small and only contains things which
Should Never Change, and all the interesting changes happen via Features.

(I am just now learning drupal and asking similar questions myself.)

Add new comment