Drupal and Composer: Managing a Drupal 8 site with Composer

The four blog posts on this topic will be as follows:

• Part 1: Understanding Composer
• Part 2: Managing a Drupal 8 site with Composer
• Part 3: Converting Management of an Existing Drupal 8 Site to Composer
• Part 4: Composer for Drupal Developers

This article will be difficult to understand without first understanding the concepts explained in part 1, so If you have not read it, it would probably be worth your while to ensure you understand the concepts outlined in the summary of that article, before proceeding with this one.

Beginning a New Project

Fortunately a lot of work has been put into creating a Composer base (called a template) for Drupal projects. This Drupal Composer template can be found on Github at: https://github.com/drupal-composer/drupal-project. Instructions on how to use the template can be found there, and the same instructions are found in the README.md file that comes with the project when installed.

Starting a new project with the Drupal Composer template can be done with the following line of code:

composer create-project drupal-composer/drupal-project:8.x-dev some-dir --stability dev --no-interaction

This command does a number of things, many of which will be addressed below.

1) A new Composer project is created

The first thing the installation does is to create the directory specified as some-dir in the command, and initializes a Composer project in that directory. Change this to the appropriate name for your project. This is now your new project. The project contains the composer.json and composer.lock files that will be used to manage the code base of the project.

2) Library dependencies (as well as their dependencies) are installed

The Composer template has a number of dependencies included by default, some of which we will take a look at here. These libraries are set by default as requirements in composer.json, and therefore are included when running the install command given earlier.

• Drush: Drush is cool. If you don’t know what it is, it’s worth some Google-fu. Anything to be written about Drush has already been written somewhere else, so check it out - it will be worth your while!
• Drupal Console: Drupal Console is also really cool. See comments on Drush.
• Composer Patches: This one is very cool. This library in and of itself is worth using Composer to manage Drupal projects in my eyes. Even if Composer had no other benefits, this one would be great. First, an explanation of a patch is necessary. A patch is kind of like a band-aid that can be applied to code. Patches allow developers to submit changes, be they bug fixes or new functionality, to the library maintainer. The library maintainer may or may not add the patch to the source code, but in the meantime, other developers can apply the patch to their own systems, both to test if it works, as well as use it if it does. However, when the library the patch has been applied to is updated to a newer version, the patches have to be re-applied. What Composer Patches does is allow developers to track patches applied to the project, and have them applied automatically during the update process. This ensures that bugs don't arise from forgetting to re-apply patches after the update. Patches are tracked by adding them to composer.json. Here is an example:
  
  "extra": {
  "patches": {
    "drupal/core”: {
      “Patch description”: "https://www.drupal.org/files/issues/someissue-1543858-30.patch"
    }
  }
}
   
• With the above code, the next time composer update drupal/core is run, Composer will attempt to apply the patch found at https://www.drupal.org/files/issues/someissue-1543858-30.patch to Drupal core. Note that the description of the patch, given above as "Patch description", is arbitrary, and should be something descriptive. If there is an issue for the patch, a link to the issue is good to add to the description, so developers can quickly look into the status of the patch, see if any updated patches have been released, and check if the patch has been incorporated into the library, rendering the patch unnecessary.
• And Others: There are many other libraries that are included as part of the Drupal Composer template, but the truth is that I haven’t looked into them. Also note that Drupal core alone has multiple dependencies which have their own dependencies. At the time of writing, 123 libraries are installed into the project with the install command.

But wait - I don’t use [fill in library here]

The Composer template is just that - a template. Some people don’t use Drush, some don’t use Drupal console. If these are not needed, they can be removed from a project in the same manner as any Composer library. Example:

composer remove drush/drush

The above command will remove Drush from the code managed by the Composer template.

3) The system folder architecture is created

The file system created by the Composer template deserves a close look.

• /web: The first thing to notice is that Drupal is installed into the /web directory in the root of the project. This means that the root of the project created with the Composer template is one level above the webroot. When configuring the server, the domain for the server will need to point at the /web directory.
• /config/sync: The Composer template sets up the project to store Drupal 8 configuration .yml files in this folder. When running drush cex sync to export configuration, the entire site configuration will be exported to this folder. This is folder is best kept out of the webroot for security purposes.
• /drush: This folder holds a few Drush specific items in it. If multiple environments exist for your project, Drush aliases can be set in /drush/sites/self.site.yml, allowing for interaction with your various environments from anywhere within the project.

• /scripts: At the time of writing, this folder contains only a single file, /scripts/composer/ScriptHandler.php. This is a really cool file that contains code run by Composer during various processes.
  
  The composer.json file in the Drupal Composer template contains the following:
   

  "scripts": {
  "drupal-scaffold": "DrupalComposer\\DrupalScaffold\\Plugin::scaffold",
  "pre-install-cmd": [
    "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
  ],
  "pre-update-cmd": [
    "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
  ],
  "post-install-cmd": [
    "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
  ],
  "post-update-cmd": [
    "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
  ]
},

  The code above executes the stated code on pre-install, pre-update, post-install and post-update. Any time either composer install or composer update are executed on the system, the pre and post hook for that call are executed, calling the relevant functions above. Developers can create their own pre/post install and update hooks following the examples shown above.

• /vendor: The first thing to note is that the location of this file differs from a vanilla Drupal 8 installation. When installing Drupal manually, the vendor folder is part of the webroot by default. This however could lead to security issues, which is why the Composer template installs it above the webroot. The vendor folder contains most of the libraries that Composer manages for your project. Drupal core, modules, themes and profiles however are saved to other locations (to be discussed in the next section). Everything else is saved to the /vendor folder.

4) Drupal File/Folder Installation Locations are set

As mentioned above, Drupal core is installed into the /web folder. The Composer template also sets up installation locations (directories) for Drupal libraries, modules, themes and profiles so that when Composer installs these, they are put in the appropriate Drupal folder locations. This is the code in composer.json that handles the installation locations:

"extra": {
  "installer-paths": {
    "web/core": [
      "type:drupal-core"
    ],
    "web/libraries/{$name}": [
      "type:drupal-library"
    ],
    "web/modules/contrib/{$name}": [
      "type:drupal-module"
    ],
    "web/profiles/contrib/{$name}": [
      "type:drupal-profile"
    ],
    "web/themes/contrib/{$name}": [
      "type:drupal-theme"
    ],
    "drush/contrib/{$name}": [
      "type:drupal-drush"
    ]
  }
}

The most common library type that a Drupal developer will install will be Drupal modules. So let’s look at the line of code specific to the module installation location:

"web/modules/contrib/{$name}": [
  "type:drupal-module"
],

This line of code says that if the type of Composer library is drupal-module, then install it to the /web/modules/contrib/[MODULE MACHINE NAME] folder.

But how does Composer know that the library being downloaded is type drupal-module? Well, the key to this is in how Composer libraries are managed in the first place. Throughout this article and the one that precedes it, we have repeatedly looked at the composer.json file that defines this project. Well, every Composer library contains a composer.json file, and every composer.json file that comes with packaged with Drupal modules contains a type declaration as follows:

"type": "drupal-module",

When Composer is installing libraries, it looks for a type declaration, and when it finds one, if there is a custom install location set in composer.json for that type, the library is installed to the declared folder. Drupal themes are of type drupal-theme, Drupal profiles are of type drupal-profile, and so on, and they are installed to the folders declared in composer.json.

Managing Custom Drupal Code with Composer

Custom code for a project can be managed with Composer as mentioned in Part 1 of this series. Drupal convention generally separates contributed (3rd party) modules and custom modules into separate folders. To install custom code in the locations according to this convention, the following lines should be added to the installer-paths declaration in composer.json:

"web/modules/custom/{$name}": [
  "type:drupal-custom-module"]
,
"web/profiles/custom/{$name}": [
  "type:drupal-custom-profile"
],
"web/themes/custom/{$name}": [
  "type:drupal-custom-theme"
],

This code adds three additional library types for custom modules, custom profiles, and custom themes.

Next, you’ll need to add a composer.json file to the custom module/theme/profile. Directions for this can be seen here: https://www.drupal.org/docs/8/creating-custom-modules/add-a-composerjson-file. Note that for the step named Define your module as a PHP package, you should set the type as drupal-custom-[TYPE], where [TYPE] is one of: module, theme, or profile.

Continuing on, make sure the composer.json file containing the type declaration has been pushed to the remote private repository.

The last step step is to add your private repository to you project’s composer.json, so that when running composer require my/privatelibrary, Composer knows in which repository to look for the library. Declaring private repositories in composer.json is explained here: https://getcomposer.org/doc/05-repositories.md#using-private-repositories.

With the above steps, when running composer install my/library, Composer will find the private repository declared in composer.json, search that repository for my/library, and download it. The composer.json file in my/library tells Composer that it’s of type drupal-custom-[TYPE], so Drupal will install it into the directory specified for Drupal custom [TYPE].

If using Git for version control on your system, you'll probably want to alter the .gitignore file in the Composer project root to ignore the custom folder locations. If you have created a custom module, and will be managing all custom modules with Composer and private repositories, you should probably add the /web/modules/custom folder to .gitignore. If you will be managing some custom modules with Git and not Composer, then you should probably add the custom module you have created to .gitignore as /web/modules/custom/[MODULE NAME].

Managing site settings: settings.php and settings.local.php

This section isn’t actually directly related to Composer and Drupal, but it’s a good step for setting up a project, and we can use the methodology to work with Composer template and Git.

Each Drupal installation depends on the file settings.php. This file is loaded as part of Drupal’s bootstrap process on every page load. Site-specific settings are added into this file, such as database connection information.

Towards the bottom of settings.php, the following lines can be found:

# if (file_exists($app_root . '/' . $site_path . '/settings.local.php')) {
# include $app_root . '/' . $site_path . '/settings.local.php';
# }

These lines are commented out by the # symbol at the start of each line. This means that the code is not executed. If these lines are uncommented, by removing the hash symbol at the start of each line, this code is executed. The code looks for a file named settings.local.php, and if it exists, it includes that file. This means any settings in settings.local.php become part of the bootstrap process and are available to Drupal. After uncommenting the code, it will look like this:

if (file_exists($app_root . '/' . $site_path . '/settings.local.php')) {
  include $app_root . '/' . $site_path . '/settings.local.php';
}

Why do this? This allows settings to be split into two types: settings that will be the same across all environments (eg. production, staging, local installations etc), and local settings that are only relevant to the current Drupal environment in which this file exists. This is done by committing settings.php to Git so it is shared amongst every environment (note - settings.local.php is NOT committed to Git, as it should not be shared). For example, the majority of Drupal installations have a separate database for each environment, meaning that database connection details will be specific to each environment. Therefore, database settings would be put into settings.local.php, as they are not shared between environments. The connection details to a remote API however may be the same regardless of environment, and therefore would be put into settings.php. This ensures any developer working on the system has access to the API details.

After splitting up the settings this way, settings.php is committed to Git so it is tracked and shared between environments. 

The full process is as follows:

1. Install the Drupal Composer template as described earlier in this article
2. Uncomment the code in settings.php as explained above
3. Install Drupal as normal
4. Run git diff settings.php to see what has been added to settings.php as part of the installation process. Anything that shows up should be added to settings.local.php, and removed from settings.php. This will definitely be the database connection, but could be other files as well.

5. Edit .gitignore in the Composer project root, and remove this line:

   /web/sites/*/settings.php

   You can now add settings for all environments to settings.php and settings specific to the local environment to settings.local.php.

Setting Up the Private Files Directory

After setting up settings.php and settings.local.php as described above, you can now add the following to settings.php:

$settings['file_private_path'] = ‘../private’;

Next, create the folder /private in the root of your Composer project. Finally clear the Drupal cache, which will create the file /private/.htaccess. At this point you can now add settings.php and the /private folder to Git. Finally, edit .gitignore and add the following:

# Ignore private files

/private/

This sets up the private file directories across all installations, saving developers having to set it up for each installation. Note that the .gitignore setting will ensure the contents of this folder are ignored by Git, as they should be.

What should I add to Git?

The Drupal Composer template project page states:

You should create a new git repository, and commit all files not excluded by the .gitignore file.

In particular, you will need to ensure you commit composer.json and composer.lock any time composer changes are made.

Is It Safe to Use Composer on a Production Server?

The actual answer to this question is not one I have. I am not a server guy overall, and for that matter, I’m not even that much of a Composer expert. It may be that Composer is entirely safe on a production server, but personally my thoughts are that having a program that can write files to the server from remote servers would seem to open up a potential security risk, and therefore it’s likely better to NOT have Composer on a production server. This comment may lead you to question why I would have wasted the time to write so much on Composer if it’s better to not use it in the first place. But not so fast partner! It really depends on the system architecture and the server setup. Some servers have been set up so that as part of the deployment process, the codebase is built using Composer, but then set up as a read-only file system or a Docker container or some other process ensuring security. This however is a particularly complex server set up. Fortunately there is an alternative for developers who are not working with servers configured in this fashion, which we'll look at next.

Using Composer, without having Composer installed on the production server

There is an in-between solution that allows us to use Composer to manage our projects, even with multiple developers, while not having Composer installed on the production server. In this case, we can use a hybrid of a Composer managed project and the old style of using pure Git for deployment.

First, we need to edit the .gitignore folder in the root of our Composer installation. In particular, we need to remove the following code:

# Ignore directories generated by Composer
/drush/contrib/
/vendor/
/web/core/
/web/modules/contrib/
/web/themes/contrib/
/web/profiles/contrib/
/web/libraries/

The above directories are all created by Composer and managed by Composer, which is why they were originally ignored by Git. However, we will not be managing our production server using Composer, and therefore we want to include these folders into the Git repository rather than ignoring them.

After setting up your project, commit the folders listed above to Git. This ensures all the files that are managed by Composer will be part of Git. That waycomposer install never needs to be run on the production server, since any code that command would download will already be part of Git.

What this means now is that any time a developer on the project add/updates code by running either composer update or composer install, they will need to commit not just the composer.json and composer.lock files, but also the added/updated source files that Composer manages, so that all code be available on the production server when checking out the code from Git.

Updating Drupal using Composer

In part one of this series, I discussed library versions. I am not going to go deep into how the versioning works internally, but I’ll explain how updating works specific to Drupal core. At the time of writing, the current version of Drupal is 8.6.3. The dependency set in composer.json is for Drupal 8.6.*. The * at the end of this means that your project uses upon any version of Drupal 8.6, so when a minor version update comes out for 8.6, for example 8.6.4, Drupal core will be updated to Drupal 8.6.4 when composer update drupal\core is run.

However, when Drupal 8.7 is released, it will not be automatically installed, since it does not fit the pattern 8.6.*. To upgrade to Drupal 8.7, the following command is run:

composer update drupal/core:~8.7

The ~/8.7 part of the above command tells Composer to use any version of Drupal 8.7. This means that in the future, when you run composer update drupal/core, minor releases of the 8.7 branch of Drupal core will be installed.

Summary

In this article, we have gone over how Composer is used with Drupal to make project deployment a little smoother, more stable, and consistent between environments. You should have an understanding of:

• How to set up a new Drupal project using Composer
• How the following folders relate to the project:
  • /config
  • /drush
  • /scripts
  • /web
  • /vendor
• How Composer adds Drupal modules and/or themes
• Drupal library dependencies
• Managing custom Drupal code with Composer
• Which items should be committed to Git
• How to use Composer to manage Drupal projects where the production server does not have Composer
• How to update Drupal using Composer

In the next post, coming soon, we'll look at how to convert an existing Drupal project to being managed by Composer.