Drupal planet https://www.morpht.com/drupal-planet/rss en Drupal 8 Architectural Paradigms: The Typed Data API https://www.morpht.com/blog/drupal-8-architectural-paradigms-typed-data-api <p><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.extra_small" data-entity-type="media" data-entity-uuid="79c2632b-5ae7-4f54-8c54-4d11d0b028ca" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-remote-video media-entity--view-mode-extra-small"><div class="field field--name-field-media-video-embed-field field--type-video-embed-field field--label-hidden field__item"><div class="video-embed-field-provider-youtube video-embed-field-responsive-video"><iframe width="854" height="480" frameborder="0" allowfullscreen="allowfullscreen" src="https://www.youtube.com/embed/TUwiQXhknq0?autoplay=0&amp;start=0&amp;rel=0"></iframe> </div> </div> </article></div> </p> Wed, 31 Jul 2019 20:22:57 +1000 Jay Friendly https://www.morpht.com/blog/drupal-8-architectural-paradigms-typed-data-api Debugging Guzzle HTTP request errors in Drupal https://www.morpht.com/blog/debugging-guzzle-http-request-errors-drupal <p>Guzzle makes HTTP requests easy. When they work, it's like magic. However, as with all coding, getting something to work requires debugging, and this is where the Drupal implementation of Guzzle has a major usability problem - any returned messages are truncated, meaning that with the default settings, error messages that can help debug an issue are not accessible to the developer. This article will show developers how they can re-structure their Guzzle queries to log the full error to the Drupal log, instead of a truncated error that does not help fix the issue.</p> <h2>Standard Methodology</h2> <p>Generally, when making a Guzzle request, it is made using a <code>try/catch</code> paradigm, so that the site does not crash in the case of an error. When not using <code>try/catch</code>, a Guzzle error will result in a WSOD, which is as bad as it gets for usability. So let's take a look at an example of how Guzzle would request a page using a standard <code>try/catch</code>:</p> <p><code>try {<br />   $client = \Drupal::httpClient();<br />   $result = $client-&gt;request('GET', 'https://www.google.com');<br /> }<br /> catch (\Exception $error) {<br />   $logger = \Drupal::logger('HTTP Client error');<br />   $logger-&gt;error($error-&gt;getMessage());<br /> }</code></p> <p>This code will request the results of <a href="http://www.google.com">www.google.com</a>, and place them in the <code>$result</code> variable. In the case that the request failed for some reason, the system logs the result of <code>$error-&gt;getMessage()</code> to the Drupal log.</p> <p>The problem, as mentioned in the intro, is that the value returned from <code>$error-&gt;getMessage()</code> contains a truncated version of the response returned from the remote website. If the developer is lucky, the text shown will contain enough information to debug the problem, but rarely is that the case. Often the error message will look something along the lines of:</p> <p><code>Client error: `POST https://exaxmple.com/3.0/users` resulted in a `400 Bad Request` response: {"type":"http://developer.example.com/documentation/guides/error-glossary/","title":"Invalid Resource","stat (truncated...)</code></p> <p>As can be seen, the full response is not shown. The actual details of the problem, and any suggestions as to a solution are not able to be seen. What we want to happen is that the full response details are logged, so we can get some accurate information as to what happened with the request.</p> <h2>Debugging Guzzle Errors</h2> <p>In the code shown above, we used the catch statement to catch <code>\Exception</code>. Generally developers will create a class that extends <code>\Exception</code>, allowing users to catch specific errors, finally catching <code>\Exception</code> as a generic default fallback.</p> <p>When Guzzle hits an error, it throws the exception <code>GuzzleHttp\Exception\GuzzleException</code>. This allows us to catch this exception first to create our own log that contains the full response from the remote server.</p> <p>We can do this, because <code>GuzzleException</code> provides the response object from the original request, which we can use to get the actual response body the remote server sent with the error. We then log that response body to the Drupal log.</p> <p><code>use Drupal\Component\Render\FormattableMarkup;<br /> use GuzzleHttp\Exception\GuzzleException;<br /> try {<br />   $response = $client-&gt;request($method, $endpoint, $options);<br /> }<br /> // First try to catch the GuzzleException. This indicates a failed response from the remote API.<br /> catch (GuzzleException $error) {<br />   // Get the original response<br />   $response = $error-&gt;getResponse();<br />   // Get the info returned from the remote server.<br />   $response_info = $response-&gt;getBody()-&gt;getContents();<br />   // Using FormattableMarkup allows for the use of &lt;pre/&gt; tags, giving a more readable log item.<br />   $message = new FormattableMarkup('API connection error. Error details are as follows:&lt;pre&gt;@response&lt;/pre&gt;', ['@response' =&gt; print_r(json_decode($response_info), TRUE)]);<br />   // Log the error<br />   watchdog_exception('Remote API Connection', $error, $message);<br /> }<br /> // A non-Guzzle error occurred. The type of exception is unknown, so a generic log item is created. catch (\Exception $error) {<br />   // Log the error.<br />   watchdog_exception('Remote API Connection', $error, t('An unknown error occurred while trying to connect to the remote API. This is not a Guzzle error, nor an error in the remote API, rather a generic local error ocurred. The reported error was @error', ['@error' =&gt; $error-&gt;getMessage()));<br /> }</code></p> <p>With this code, we have caught the Guzzle exception, and logged the actual content of the response from the remote server to the Drupal log. If the exception thrown was any other kind of exception than <code>GuzzleException</code>, we are catching the generic <code>\Exception</code> class, and logging the given error message.</p> <p>By logging the response details, our log entry will now look something like this:</p> <p><code>Remote API connection error. Error details are as follows:</code></p> <p><code>stdClass Object (<br />   [title] =&gt; Invalid Resource<br />   [status] =&gt; 400<br />   [detail] =&gt; The resource submitted could not be validated. For field-specific details, see the 'errors' array.<br />   [errors] =&gt; Array (<br />     [0] =&gt; stdClass Object (<br />       [field] =&gt; some_field<br />       [message] =&gt; Data presented is not one of the accepted values: 'Something', 'something else', or another thing'<br />     )<br />   )<br /> )</code></p> <p>* Note that this is just an example, and that each API will give its own response structure.</p> <p>This is a much more valuable debug message than the original truncated message, which left us understanding that there had been an error, but without the information required to fix it.</p> <h2>Summary</h2> <p>Drupal 8 ships with Guzzle, an excellent HTTP client for making requests to other servers. However, the standard debugging method doesn't provide a helpful log message from Guzzle. This article shows how to catch Guzzle errors, so that the full response can be logged, making debugging of connection to remote servers and APIs much easier.</p> <p>Happy Drupaling!</p> <div class="display-xs">Image credit: Marco Verch <a href="https://www.flickr.com/photos/30478819@N08/35337269544">https://www.flickr.com/photos/30478819@N08/35337269544</a></div> Thu, 06 Jun 2019 03:09:51 +1000 Jay Friendly https://www.morpht.com/blog/debugging-guzzle-http-request-errors-drupal Drupal 8 Configuration - Module developers and the API https://www.morpht.com/blog/drupal-8-configuration-part-5-module-developers-and-api <h2>Background</h2> <p>We live in an age of Drupal complexity. In the early days of Drupal, many developers would have a single Drupal instance/environment (aka copy) that was their production site, where they would test out new modules and develop new functionality. Developing on the live website however sometimes met with disastrous consequences when things went wrong! Over time, technology on the web grew, and nowadays it's fairly standard to have a Drupal project running on multiple environments to allow site development to be run in parallel to a live website without causing disruptions. New functionality is developed first in isolated private copies of the website, put into a testing environment where it is approved by clients, and eventually merged into the live production site.</p> <p>While multiple environments allow for site development without causing disruptions on the live production website, it introduces a new problem; how to ensure consistency between site copies so that they are all working with the correct code.</p> <p>This series of articles will explore the Configuration API, how it enables functionality to be migrated between multiple environments (sites), and ways of using the Configuration API with contributed modules to effectively manage the configuration of a project. This series will consist of the following posts:</p> <ul><li>Part 1: <a href="/blog/drupal-8-configuration-part-1-configuration-api">The Configuration API</a></li> <li>Part 2: <a href="/blog/drupal-8-configuration-part-2-how-api-works" hreflang="en">How the API works</a></li> <li>Part 3: <a href="/blog/drupal-8-configuration-part-3-using-api">Using the API</a></li> <li>Part 4: <a href="/blog/drupal-8-configuration-extending-api">Extending the API with contributed modules</a></li> <li>Part 5: <strong>Module developers and the API</strong></li> </ul><p>This article will focus specifically on how developers can manage, declare, and debug configuration in their custom modules.</p> <h2>Configuration Schema</h2> <p>Configuration schema describes the type of configuration a module introduces into the system. Schema definitions are used for things like translating configuration and its values, for typecasting configuration values into their correct data types, and for migrating configuration between systems. Having configuration in the system is not as helpful without metadata that describes what the configuration is. Configuration schemas define the configuration items.</p> <p>Any module that introduces any configuration into the system MUST define the schema for the configuration the module introduces.</p> <p>Configuration schema definitions are declared in <code>[MODULE ROOT]/config/schema/[MODULE NAME].schema.yml</code>, where <code>[MODULE NAME]</code> is the machine name of the module. Schema definitions may define one or multiple configuration objects. Let's look at the configuration schema for the <a href="https://www.drupal.org/project/restrict_ip">Restrict IP module</a> for an example. This module defines a single configuration object, <code>restrict_ip.settings</code>:</p> <p><code>restrict_ip.settings:<br />   type: config_object<br />   label: 'Restrict IP settings'<br />   mapping:<br />     enable:<br />       type: boolean<br />       label: 'Enable module'<br />     mail_address:<br />       type: string<br />       label: 'Contact mail address to show to blocked users'<br />     dblog:<br />       type: boolean<br />       label: 'Log blocked access attempts'<br />     allow_role_bypass:<br />       type: boolean<br />       label: 'Allow IP blocking to be bypassed by roles'<br />     bypass_action:<br />       type: string<br />       label: 'Action to perform for blocked users when bypassing by role is enabled'<br />     white_black_list:<br />       type: integer<br />       label: 'Whether to use a path whitelist, blacklist, or check all pages'<br />     country_white_black_list:<br />       type: integer<br />       label: 'Whether to use a whitelist, blacklist, or neither for countries'<br />     country_list:<br />       type: string<br />       label: 'A colon separated list of countries that should be white/black listed'</code></p> <p>The above schema defines the config object <em>restrict_ip.settings</em> which is of type <em>config_object</em> (defined in <a href="https://cgit.drupalcode.org/drupal/tree/core/config/schema/core.data_types.schema.yml#n108">core.data_types.schema.yml</a>).</p> <p>When this module is enabled, and the configuration is exported, the filename of the configuration will be <code>restrict_ip.settings.yml</code>. This object has the keys <code>enable</code>, <code>mail_address</code>, <code>dblog</code> etc. The schema tells what type of value is to be stored for each of these keys, as well as the label of each key. Note that this label is automatically provided to Drupal for translation.</p> <p>The values can be retrieved from the <code>restrict_ip.settings</code> object as follows:</p> <p><code>$enable_module = \Drupal::config('restrict_ip.settings')-&gt;get('enable');<br /> $mail_address = \Drupal::config('restrict_ip.settings')-&gt;get('mail_address');<br /> $log = \Drupal::config('restrict_ip.settings')-&gt;get('dblog');</code></p> <p>Note that modules defining custom fields, widgets, and/or formatters must define the schema for those plugins. See <a href="https://www.drupal.org/node/2381105">this page</a> to understand how the schema definitions for these various plugins should be defined.</p> <h3>Default configuration values</h3> <p>If configuration needs to have default values, the default values can be defined in <code>[MODULE ROOT]/config/install/[CONFIG KEY].yml</code> where <code>[CONFIG KEY]</code> is the configuration object name. Each item of configuration defined in the module schema requires its own YML file to set defaults. In the case of the Restrict IP module, there is only one config key, <code>restrict_ip.settings</code>, so there can only be one file to define the default configuration, <code>restrict_ip/config/install/<strong>restrict_ip.settings</strong>.yml</code>. This file will then list the keys of the configuration object, and the default values. In the case of the Restrict IP module, the default values look like this:</p> <p><code>enable: false<br /> mail_address: ''<br /> dblog: false<br /> allow_role_bypass: false<br /> bypass_action: 'provide_link_login_page'<br /> white_black_list: 0<br /> country_white_black_list: 0<br /> country_list: ''</code><br />  </p> <p>As can be seen, each of the mapped keys of the <code>restrict_ip.settings</code> config_object in the schema definition are added to this file, with the default values provided for each key. If a key does not have a default value, it can be left out of this file. When the module is enabled, these are the values that will be imported into active configuration as defaults.</p> <h2>Debugging Configuration</h2> <p>When developing a module, it is important to ensure that the configuration schema accurately describes the configuration used in the module. Configuration can be inspected using the <a href="https://www.drupal.org/project/config_inspector">Configuration Inspector module</a>. After enabling your custom module, visit the reports page for the Configuration Inspector at /admin/reports/config-inspector, and it will list any errors in configuration.</p> <p><figure role="group" class="caption caption-drupal-entity"><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.full" data-entity-type="media" data-entity-uuid="47830d29-7c03-42df-b3b6-063183834da6" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-image media-entity--view-mode-full"><div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <picture><!--[if IE 9]><video style="display: none;"><![endif]--><source srcset="/sites/morpht/files/styles/full_lg/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=8bi9tjrF 1x, /sites/morpht/files/styles/full_lg_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=jKZIj9JL 2x" media="(min-width: 992px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_md/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=2Z_WEDnI 1x, /sites/morpht/files/styles/full_md_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=irEnvrAH 2x" media="(min-width: 768px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_sm/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=ux89Klyf 1x, /sites/morpht/files/styles/full_sm_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=ZnwsZAL1 2x" media="(min-width: 576px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=xv_X0VAZ 1x, /sites/morpht/files/styles/full_xs_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=YvWn56sr 2x" type="image/png"></source><!--[if IE 9]></video><![endif]--><img src="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20configuration%20inspector%20error%20item.png?itok=xv_X0VAZ" alt="Screenshot of an item with an error in the configuration inspector" typeof="foaf:Image" /></picture></div> </article></div> <figcaption>The Configuration Inspector module errors in configuration schema definitions</figcaption></figure></p> <p>Clicking on 'List' for items with errors will give more details as to the error.</p> <p><figure role="group" class="caption caption-drupal-entity"><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.full" data-entity-type="media" data-entity-uuid="47a4cf27-2057-4a4a-ab23-00e5d7338773" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-image media-entity--view-mode-full"><div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <picture><!--[if IE 9]><video style="display: none;"><![endif]--><source srcset="/sites/morpht/files/styles/full_lg/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=bSlJWubv 1x, /sites/morpht/files/styles/full_lg_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=E64wzKJj 2x" media="(min-width: 992px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_md/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=Eyk-3-vt 1x, /sites/morpht/files/styles/full_md_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=jFZ6M5WV 2x" media="(min-width: 768px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_sm/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=7t6JTXsy 1x, /sites/morpht/files/styles/full_sm_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=qY30A21t 2x" media="(min-width: 576px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=slKuEsA- 1x, /sites/morpht/files/styles/full_xs_hi/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=-MqdZoJf 2x" type="image/png"></source><!--[if IE 9]></video><![endif]--><img src="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20configuration%20inspector%20expanded%20error%20item.png?itok=slKuEsA-" alt="Screenshot of the Configuration Inspector module showing a configuration item with an error" typeof="foaf:Image" /></picture></div> </article></div> <figcaption>The 'enable' key has an error in schema. The stored value is a boolean, but the configuration definition defines a string</figcaption></figure></p> <p>Using the Configuration Inspector module, you can find where you have errors in your configuration schema definitions. Cleaning up these errors will correctly integrate your module with the Configuration API. In the above screenshot, then type of data in the active schema is a boolean, yet the configuration schema defines it as a string. The solution is to change the schema definition to be a boolean.</p> <h2>Summary</h2> <p>In this final article of this series on the Drupal 8 Configuration API, we looked at configuration schema, how developers can define this schema in their modules and provide defaults, as well as how to debug configuration schema errors. Hopefully this series will give you a fuller understanding of what the Configuration API is, how it can be managed, and how you can use it effectively in your Drupal projects. Happy Drupaling!</p> Thu, 16 May 2019 15:45:44 +1000 Jay Friendly https://www.morpht.com/blog/drupal-8-configuration-part-5-module-developers-and-api Drupal 8 Configuration - Extending the API https://www.morpht.com/blog/drupal-8-configuration-part-4-extending-api-contributed-modules <h2>Background</h2> <p>We live in an age of Drupal complexity. In the early days of Drupal, many developers would have a single Drupal instance/environment (aka copy) that was their production site, where they would test out new modules and develop new functionality. Developing on the live website however sometimes met with disastrous consequences when things went wrong! Over time, technology on the web grew, and nowadays it's fairly standard to have a Drupal project running on multiple environments to allow site development to be run in parallel to a live website without causing disruptions. New functionality is developed first in isolated private copies of the website, put into a testing environment where it is approved by clients, and eventually merged into the live production site.</p> <p>While multiple environments allow for site development without causing disruptions on the live production website, it introduces a new problem; how to ensure consistency between site copies so that they are all working with the correct code.</p> <p>This series of articles will explore the Configuration API, how it enables functionality to be migrated between multiple environments (sites), and ways of using the Configuration API with contributed modules to effectively manage the configuration of a project. This series will consist of the following posts:</p> <ul><li>Part 1: <a href="/blog/drupal-8-configuration-part-1-configuration-api">The Configuration API</a></li> <li>Part 2: <a href="/blog/drupal-8-configuration-part-2-how-api-works" hreflang="en">How the API works</a></li> <li>Part 3: <a href="/blog/drupal-8-configuration-part-3-using-api">Using the API</a></li> <li><strong>Part 4: Extending the API with contributed modules</strong></li> <li>Part 5: <a href="/node/491">Module developers and the API</a></li> </ul><p><a href="/blog/drupal-8-configuration-part-1-configuration-api">Part 1</a> gives the background of the Configuration API, as well as discusses some terminology used within this article, and <a href="/blog/drupal-8-configuration-part-2-how-api-works">Part 2</a> describes how the API works, and <a href="/blog/drupal-8-configuration-part-3-using-api">Part 3</a> explains how to use functionality provided by core, so they are worth a read before beginning this article. </p> <h2>Read-only configuration</h2> <p>In some situations, site builders may want to prevent any configuration changes from being made on the production environment, preventing changes that may cause unexpected issues. For example, clients with admin access could log into the production server, and make what they think is an innocent configuration change, that results in unexpected and drastic consequences. Some site builders consider it to be a best practice to prevent configuration changes on the production server altogether, under the idea that only content should be editable on the production server, and configuration changes should only be made in development and/or staging environments before being tested and pushed to production.</p> <p>The <a href="https://www.drupal.org/project/config_readonly">Config Readonly module</a>, allows for configuration changes through the UI to be disabled on a given environment. It does this by disabling the submit buttons on configuration pages. The module also disables configuration changes using Drush and Drupal console.</p> <p><figure role="group" class="caption caption-drupal-entity"><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.full" data-entity-type="media" data-entity-uuid="0d16b1e4-ac66-46da-b07a-e3da3c58b1ca" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-image media-entity--view-mode-full"><div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <picture><!--[if IE 9]><video style="display: none;"><![endif]--><source srcset="/sites/morpht/files/styles/full_lg/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=5BNmwXO2 1x, /sites/morpht/files/styles/full_lg_hi/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=7CCuz8kt 2x" media="(min-width: 992px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_md/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=FER8l3x2 1x, /sites/morpht/files/styles/full_md_hi/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=3yKDm8_0 2x" media="(min-width: 768px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_sm/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=LZbpcx87 1x, /sites/morpht/files/styles/full_sm_hi/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=vGtbAl9i 2x" media="(min-width: 576px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=3uC9wDN_ 1x, /sites/morpht/files/styles/full_xs_hi/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=PC8ofNUU 2x" type="image/png"></source><!--[if IE 9]></video><![endif]--><img src="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20config%20readonly%20module.png?itok=3uC9wDN_" alt="Screenshot of the Configuration Readonly module output on configuration pages" typeof="foaf:Image" /></picture></div> </article></div> <figcaption>A configuration form that has been disabled with the Configuration Readonly module</figcaption></figure></p> <p>Note: some configuration forms may still be enabled when using this  module. Module developers must build their forms by extending ConfigFormBase for the Configuration Readonly module to do its magic. If the developer has built the form using other means, the form will not be disabled, and the configuration for that form can be changed through the admin UI.</p> <p>To set up an environment as read-only, add the following line to <code>settings.php</code>, then enable the module:</p> <p><code>$settings['config_readonly'] = TRUE; </code></p> <p>After an environment is set as read-only, changes to configuration can only be made on other environments, then migrated and imported into the active configuration on the read-only environment.</p> <h3 id="blacklist_configuration">Complete split (blacklist) configuration</h3> <p>Sometimes configuration needs to exist on some environments, but not exist in other environments. For example, development modules, like the <a href="https://www.drupal.org/project/devel">Devel module</a>, or UI modules like Views UI (Drupal core) and Menu UI (Drupal core) should not be enabled on production environments, as they add overhead to the server while being unnecessary since the production server should not be used for development.</p> <p>A problem arises when configuration is exported from one environment, and imported into the production environment. All the configuration from the source environment is now the active configuration on the production environment. So any development modules that were enabled on the source environment are now enabled on the production environment. In the case of development modules like Devel, this may only add some overhead to the server, but imagine a module like the <a href="https://www.drupal.org/project/shield">Shield module</a>, which sets up environments to require a username and password before even accessing the site. If this module is accidentally enabled upon import on production, it will block the site from public access - a disaster!</p> <p>The solution to this situation is to <em>blacklist</em> configuration. Blacklisted configuration is blacklisted (removed) from configuration upon export. This functionality is provided by the <a href="https://www.drupal.org/project/config_split">Configuration Split module</a>. This module allows for black-listing configuration either by module, by individual configuration key(s), and/or by wildcard.</p> <p>Note that more detailed directions for creating blacklists can be found on the <a href="https://www.drupal.org/docs/8/modules/configuration-split/creating-a-simple-split-configuration-dev-modules-only-in-dev">documentation page</a>. The following is meant to give an overview of how black lists work.</p> <p>Blacklists are created as part of a configuration profile. Configuration profiles allow for 'splitting' (a divergence in) configuration between environments. Profiles may be created for environment types such <em>development</em>, <em>staging</em> and <em>production</em> allowing for configuration specific to those types of environments. Or profiles could be set up for<em> public non-production</em> environments, that would have the Shield module enabled and configured. While a <em>development</em> profile may apply to all development environments, not all development environments are on publicly accessible URLs, and therefore may not need the Shield module enabled.</p> <p>When setting up a configuration profile, note that the folder name must be the same as the <em>machine_name</em> of the profile.</p> <p><figure role="group" class="caption caption-drupal-entity"><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.full" data-entity-type="media" data-entity-uuid="67934ec9-a3de-4526-a40d-8809b88b62ed" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-image media-entity--view-mode-full"><div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <picture><!--[if IE 9]><video style="display: none;"><![endif]--><source srcset="/sites/morpht/files/styles/full_lg/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=2iG9AiW0 1x, /sites/morpht/files/styles/full_lg_hi/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=GEc0V_zU 2x" media="(min-width: 992px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_md/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=d80UVDpY 1x, /sites/morpht/files/styles/full_md_hi/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=O0opSBR9 2x" media="(min-width: 768px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_sm/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=JgKyaUV6 1x, /sites/morpht/files/styles/full_sm_hi/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=k-_nXvTN 2x" media="(min-width: 576px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=fe4xPph0 1x, /sites/morpht/files/styles/full_xs_hi/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=fX8M7ogX 2x" type="image/png"></source><!--[if IE 9]></video><![endif]--><img src="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20Config%20Split%20Profile.png?itok=fe4xPph0" alt="Screenshot of configuration split profile settings" typeof="foaf:Image" /></picture></div> </article></div> <figcaption>Configuration split profile settings</figcaption></figure></p> <p>Note that you must manually create the folder specified above, and that folder can and should be tracked using Git, so it can be use on any environment that enables the profile.</p> <p>Configuration can then be set up to be blacklisted either by module, by configuration key, or by wildcard:</p> <p><figure role="group" class="caption caption-drupal-entity"><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.full" data-entity-type="media" data-entity-uuid="36c47360-ecb6-4520-9d31-0b19eddf007b" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-image media-entity--view-mode-full"><div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <picture><!--[if IE 9]><video style="display: none;"><![endif]--><source srcset="/sites/morpht/files/styles/full_lg/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=v6x9_XQS 1x, /sites/morpht/files/styles/full_lg_hi/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=EIOW0XyW 2x" media="(min-width: 992px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_md/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=8mYdOe1O 1x, /sites/morpht/files/styles/full_md_hi/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=xeqg0zF5 2x" media="(min-width: 768px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_sm/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=5n3f2NVv 1x, /sites/morpht/files/styles/full_sm_hi/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=y2Pi6-Lp 2x" media="(min-width: 576px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=5_DwCQTW 1x, /sites/morpht/files/styles/full_xs_hi/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=_PSw4q_H 2x" type="image/png"></source><!--[if IE 9]></video><![endif]--><img src="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20conflig%20split%20complete%20split%20settings.png?itok=5_DwCQTW" alt="Screenshot of configuration split complete-split/blacklist settings" typeof="foaf:Image" /></picture></div> </article></div> <figcaption>Complete split (blacklist) can be set by module, configuration key, or by wildcard</figcaption></figure></p> <p>Finally, environments need to be set up to use a given profile. This is handled by adding the following line to settings.php on the environment:</p> <p><code>$config['config_split.config_split.PROFILEMACHINENAME']['status'] = TRUE;</code></p> <p>Where <code>PROFILEMACHINENAME</code> is the machine_name from the profile you created.</p> <p>Although blacklisted configuration does not become part of the exported archive, it is not ignored altogether. When an environment has the profile enabled, upon export, blacklisted configuration is extracted, then written to the folder specified in the profile. The remaining configuration is written to the default configuration directory. When importing configuration, environments with the profile enabled will first retrieve the configuration from the default configuration directory, then apply any configuration from the folder specified in the profile. Environments not set up to use the profile ignore the configuration in the blacklisted directory altogether on both import and export.</p> <p>This means that a developer can enable the Devel module on their local environment, blacklist it, then export their configuration. The blacklisted configuration never becomes part of the default configuration, and therefore the module will not accidentally be installed on environments with the configuration profile enabled.</p> <h3>Conditional split (grey list) configuration</h3> <p>Grey lists, also provided by the <a href="https://www.drupal.org/project/config_split">Configuration Split module</a>, allow for configuration to differ by environment. With a blacklist (previous section), the configuration only exists in the active database configuration for environments that are set up to use the configuration profile containing the blacklisted configuration. With a grey list, the configuration exists in the active configuration in all environments, but the configuration profiles can be set up to allow environments to use differing values for the configuration.</p> <p><span>Imagine an integration with a remote API requiring a username, password, and endpoint URL. The production server needs integrate with the remote API's production instance, while other environments will integrate with the remote API's sandbox instance. As such, the values to be used will differ by environment:</span></p> <p><u>Production Environment:</u></p> <p>remoteapi.username: <em>ProductionUsername</em><br /> remoteapi.password: <em>ProductionPassword</em><br /> remoteapi.endpoint:<em> <a href="https://example.com/api">https://example.com/api</a></em></p> <p><u>Other Environments:</u></p> <p>remoteapi.username: <em>SandboxUsername</em><br /> remoteapi.password: Sandbox<em>Password</em><br /> remoteapi.endpoint:<em> <a href="https://sandbox.example.com/api">https://sandbox.example.com/api</a></em></p> <p><span>A grey list allows for the setup of these values by configuration profile.</span></p> <p><span>You may be remembering that </span><a href="/blog/drupal-8-configuration-part-3-using-api">Part 3</a><span> of this series of articles discussed overriding configuration in </span><code>settings.php</code><span>, and thinking that a grey list sounds like the same thing. After all, the default values for the sandbox instance of the API could be set up as the configuration values, and the production values could be overridden in </span><code>settings.php</code><span> on the production environment, with the same end-result.</span></p> <p><span>The difference is that with a grey list, the remote API values are saved to the configuration profile folder, which is tracked by Git, and therefore can be tracked and migrated between environments. </span>When grey listed configuration is exported, the grey listed configuration is written to the configuration profile folder, in the same manner as blacklisted configuration. When configuration is imported, the default values are retrieved, and the grey list values are used to override the default values, after which the configuration is imported into active configuration.</p> <p>With the configuration override method using <code>settings.php</code>, site builders need to store the various configuration values somewhere outside the project, communicating environment-specific configuration values to each other through some means, to be manually entered on the relevant environment(s). With a grey list, the configuration values are managed with Git, meaning site builders do not need to record them outside the project, nor communicate them to each other through some other means. Site builders simply need to enable the relevant configuration profile in <code>settings.php</code>, and the environment-specific values can then be imported into active configuration from the configuration profile directory. This means that the sandbox API values can be set up as the values used by default on all environments, and a production configuration profile can be enabled on the production environment using the values to connect to the production instance of the remote API.</p> <p>Conditional split items can be selected either from a list, or by manually entering them into the configuration profile:</p> <p><figure role="group" class="caption caption-drupal-entity"><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.full" data-entity-type="media" data-entity-uuid="29ffbf9c-3dd1-4c77-a583-93c3494cc2fb" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-image media-entity--view-mode-full"><div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <picture><!--[if IE 9]><video style="display: none;"><![endif]--><source srcset="/sites/morpht/files/styles/full_lg/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=7X2a-8ns 1x, /sites/morpht/files/styles/full_lg_hi/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=e2HpyJkU 2x" media="(min-width: 992px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_md/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=9VYkhdZc 1x, /sites/morpht/files/styles/full_md_hi/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=sHP5uLrH 2x" media="(min-width: 768px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_sm/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=WGSDN3l_ 1x, /sites/morpht/files/styles/full_sm_hi/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=IoLy_X9O 2x" media="(min-width: 576px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=Z4o-VbDY 1x, /sites/morpht/files/styles/full_xs_hi/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=CAAQzmNV 2x" type="image/png"></source><!--[if IE 9]></video><![endif]--><img src="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20config%20split%20grey%20list%20settings.png?itok=Z4o-VbDY" alt="Screenshot of configuration split conditional-split/grey list settings" typeof="foaf:Image" /></picture></div> </article></div> <figcaption>Conditional split (grey list) settings can be selected or manually entered</figcaption></figure></p> <p>Finally, n<span>ote that grey lists can actually be used in conjunction with configuration overrides in </span><code>settings.php</code><span>. Grey lists are applied during import and export of configuration from the database. Values in </span><code>settings.php</code><span> are used at runtime, overriding any active configuration. So a developer could choose to set up their local instance of the system to connect to an entirely different instance of the remote API altogether by overriding the values in </span><code>settings.php</code><span>.</span></p> <h3>Ignoring configuration (overwrite protection)</h3> <p>Sometimes developers will want to protect certain configuration items in the database from ever being overwritten. For example imagine a site named <em>Awesome Site</em>, with a module that supplies the core of the site, named <code>awesome_core</code>. Since this module provides the core functionality of the site, it should never be disabled under any circumstances, as that would disable the core functionality of the site. In this case, the configuration for this module can be set to be 'ignored'. Any attempts to import ignored configuration from the file system to the active configuration in database will be skipped, and not imported.</p> <p>Configuration can be ignored using the <a href="https://www.drupal.org/project/config_ignore">Config Ignore module</a>. T<span>he functionality this module provides is similar to the functionality provided by the Config Readonly module discussed earlier, however the Config Readonly module covers the entire configuration of an environment, while the Config Ignore module allows for choosing configuration that should be protected. This configuration is protected by ignoring it altogether on import.</span></p> <p>Configuration can be ignored as follows:</p> <ol><li>Enable <a href="https://www.drupal.org/project/config_ignore">Config Ignore module</a> on all environments.</li> <li>Navigate to the config ignore UI page, and set the configuration item to be ignored. In the case of preventing the awesome_core module from being disabled, the following would be added:<br /><code>core.extension:module.awesome_core</code><br /><figure role="group" class="caption caption-drupal-entity"><div data-embed-button="media_entity_embed" data-entity-embed-display="view_mode:media.full" data-entity-type="media" data-entity-uuid="6bbedc6f-b8ea-421b-bb3a-b853f3a66864" data-langcode="en" class="embedded-entity"><article class="media-entity media-entity--type-image media-entity--view-mode-full"><div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <picture><!--[if IE 9]><video style="display: none;"><![endif]--><source srcset="/sites/morpht/files/styles/full_lg/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=oJTE12b7 1x, /sites/morpht/files/styles/full_lg_hi/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=3JbR3mR8 2x" media="(min-width: 992px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_md/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=TJNce8BC 1x, /sites/morpht/files/styles/full_md_hi/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=ONeqwar- 2x" media="(min-width: 768px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_sm/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=qQR4kv45 1x, /sites/morpht/files/styles/full_sm_hi/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=s-0yoAgH 2x" media="(min-width: 576px)" type="image/png"></source><source srcset="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=posxd_Xc 1x, /sites/morpht/files/styles/full_xs_hi/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=wo3H9dKT 2x" type="image/png"></source><!--[if IE 9]></video><![endif]--><img src="/sites/morpht/files/styles/full_xs/public/2019-03/screenshot%20-%20configuration%20ignore%20module%20admin%20page.png?itok=posxd_Xc" alt="Screenshot of the Configuration Ignore module admin page" typeof="foaf:Image" /></picture></div> </article></div> <figcaption>Configuration to be ignore is entered one item per line. Wildcards can be used.</figcaption></figure></li> </ol><p>This setting will ensure that any attempts to change or remove core.extension:module.awesome_core upon configuration import will be ignored. So if the module is enabled on production, and a developer pushes configuration changes that would uninstall this module, those changes will be ignored, and the module will still be set as enabled after import.</p> <h2>Summary</h2> <p>In this article, we looked at various modules that extend the Configuration API, use cases behind these modules, and how the modules worked. We looked at the <a href="https://www.drupal.org/project/config_readonly">Config Readonly module</a>, the <a href="https://www.drupal.org/project/config_split">Configuration Split module</a>, and the <a href="https://www.drupal.org/project/config_ignore">Config Ignore module</a>, and how to use these modules to manage configuration differences between environments. In the next, final <a href="/blog/drupal-8-configuration-part-5-module-developers-and-api">fifth part</a> of this series, we will look at configuration management for module developers, and how developers can define the schema for the configuration in modules they develop.</p> Wed, 15 May 2019 13:41:44 +1000 Jay Friendly https://www.morpht.com/blog/drupal-8-configuration-part-4-extending-api-contributed-modules Drupal 8 Configuration - Part 2: How the API works https://www.morpht.com/blog/drupal-8-configuration-part-2-how-api-works <h2>Background</h2> <p>We live in an age of Drupal complexity. In the early days of Drupal, many developers would have a single Drupal instance/environment (aka copy) that was their production site, where they would test out new modules and develop new functionality. Developing on the live website however sometimes met with disastrous consequences when things went wrong! Over time, technology on the web grew, and nowadays it's fairly standard to have a Drupal project running on multiple environments to allow site development to be run in parallel to a live website without causing disruptions. New functionality is developed first in isolated private copies of the website, put into a testing environment where it is approved by clients, and eventually merged into the live production site.</p> <p>While multiple environments allow for site development without causing disruptions on the live production website, it introduces a new problem; how to ensure consistency between site copies so that they are all working with the correct code.</p> <p>This series of articles will explore the Configuration API, how it enables functionality to be migrated between multiple environments (sites), and ways of using the Configuration API with contributed modules to effectively manage the configuration of a project. This series will consist of the following posts:</p> <ul><li>Part 1: <a href="/blog/drupal-8-configuration-part-1-configuration-api">The Configuration API</a></li> <li><strong><a hreflang="en">Part 2: How the API works</a></strong></li> <li>Part 3: <a href="/node/486">Using the API</a></li> <li>Part 4: <a href="/node/496">Extending the API with contributed modules</a></li> <li>Part 5: <a href="/node/491">Module developers and the API</a></li> </ul><p><a href="/blog/drupal-8-configuration-part-1-configuration-api">Part 1</a> gives the background of the Configuration API, as well as discusses some terminology used within this article, so it's worth a read before beginning this article.</p> <h2>Active configuration is in the database</h2> <p>In Drupal 8, configuration used at runtime is stored in the database. The values in the database are known as <em>active configuration</em>. In Drupal 7, configuration was known as settings, and stored in the <code>{variable}</code> table. In Drupal 8, configuration is stored in the <code>{config}</code> table. The active configuration is used at runtime by Drupal when preparing responses.</p> <h2>Configuration is backed up to files</h2> <p>The Configuration API enables the ability to export the active database configuration into a series of YML files. These files can also be imported into the database. This means that a developer can create a new Field API field on their local development environment, export the configuration for the new field to files, push those files to to the production environment, then import the configuration into the production environment's active configuration in the database.</p> <p>The configuration values in the database are the live/active values, used by Drupal when responding to requests. The YMLfiles that represent configuration are not required, and are not used at run-time. In fact, in a new system the configuration files don't even exist until/unless someone exports the active configuration from the database. The configuration files are a means to be able to back up and/or migrate configuration between environments. Configuration files are never used in runtime on a site.</p> <h2>Configuration architecture</h2> <p>Let's look at the Configuration API on a more technical level, using a real-world example. The <a href="https://www.drupal.org/project/restrict_ip">Restrict IP</a> module allows users to set a list of rules that whitelist or blacklist users based on their IP address. Upon visiting the module settings page, users are presented with a checkbox that allows them to enable/disable the module functionality.</p> <p>From a data standpoint, checkboxes are booleans; they represent either a <code>true</code> or <code>false</code> value. When exporting the configuration of a site with the Restrict IP module enabled, the relevant configuration key will be saved with a value of either <code>true</code> or <code>false</code> to a .yml file. Modules are required to define the schema for any configuration the module creates. Developers can look at the configuration schema declarations to understand what file(s) will be created, and what values are accepted.</p> <p>Modules declare the schema for their configuration in the <code>[MODULE ROOT]/config/schema</code> directory. In the case of the Restrict IP module, the schema file is <code>restrict_ip/config/schema/restrict_ip.schema.yml</code>. This file contains the following declaration:</p> <p><code>restrict_ip.settings:<br />   type: config_object<br />   label: 'Restrict IP settings'<br />   mapping:<br />     enable:<br />       type: boolean<br />       label: 'Enable module'</code></p> <p>Schema declarations tell the system what the configuration looks like. In this case, the base configuration object is <code>restrict_ip.settings</code>, from the first line. When this configuration is exported to file, the file name will be <code>restrict_ip.settings.yml</code>. In that file will be a declaration of either:</p> <p><code>enable: true</code></p> <p>Or:</p> <p><code>enable: false</code></p> <p>When the file <code>restrict_ip.settings.yml</code> is imported into the active configuration in another environment's database, the value for the <code>enable</code> key will be imported as defined in the file.</p> <p>On top of this, enabled modules are listed in <code>core.extension.yml</code>, which is the configuration that tracks which modules are enabled in a given environment. When the Restrict IP module is enabled in one environment, and configuration files exported from that environment are imported into a different Drupal environment, the Restrict IP module will be enabled due to its existence in <code>core.extension.yml</code>, and the setting <code>enable</code> will have a value of either <code>true</code> or <code>false</code>, depending on what the value was exported from the original environment.</p> <p>Note that if you were to try to import the configuration without having the Restrict IP module in the codebase, an error will be thrown and the configuration import will fail with an error about the Restrict IP module not existing.</p> <h2>Summary</h2> <p>In this article, we looked at how the Drupal 8 Configuration API works on a technical level. We looked at how active configuration lives in the database, and can be exported to files which can then be imported back into the database, or migrated and imported to other Drupal environments. In <a href="/blog/drupal-8-configuration-part-3-using-api">part 3</a> of the series, Using the API, we will look at how to actually use the Configuration API, as well as some contributed modules that extend the functionality of the Configuration API, allowing for more effective management of Drupal 8 projects.</p> Mon, 13 May 2019 12:19:46 +1000 Jay Friendly https://www.morpht.com/blog/drupal-8-configuration-part-2-how-api-works Drupal 8 Configuration - Part 1: The Configuration API https://www.morpht.com/blog/drupal-8-configuration-part-1-configuration-api <h2>Background</h2> <p>We live in an age of Drupal complexity. In the early days of Drupal, many developers would have a single Drupal instance/environment (aka copy) that was their production site, where they would test out new modules and develop new functionality. Developing on the live website however sometimes met with disastrous consequences when things went wrong! Over time, technology on the web grew, and nowadays it's fairly standard to have a Drupal project running on multiple environments to allow site development to be run in parallel to a live website without causing disruptions. New functionality is developed first in isolated private copies of the website, put into a testing environment where it is approved by clients, and eventually merged into the live production site.</p> <p>While multiple environments allow for site development without causing disruptions on the live production website, it introduces a new problem; how to ensure consistency between site copies so that they are all working with the correct code.</p> <p>This series of articles will explore the Configuration API, how it enables functionality to be migrated between multiple environments (sites), and ways of using the Configuration API with contributed modules to effectively manage the configuration of a project. This series will consist of the following posts:</p> <ul><li><strong>Part 1: The Configuration API</strong></li> <li>Part 2: <a href="https://www.morpht.com/blog/drupal-8-configuration-part-2-how-api-works">How the API works</a></li> <li>Part 3: <a href="/node/486">Using the API</a></li> <li>Part 4: <a href="/node/496">Extending the API with contributed modules</a> </li> <li>Part 5: <a href="/node/491">Module developers and the API</a></li> </ul><h2>Terminology</h2> <p>Before we get started, let's review some terminology. A Drupal <em>project</em> is the overall combination of the core codebase and all of the <em>environments</em> that are used for development of the project. An <em>environment</em> is a copy/clone of the website that is accessible at a unique domain name (URL). When most users on the web think of a website, they think of the <em>environment</em> accessed at a single URL, like google.com or facebook.com. These URLs are the entry to the live production <em>environments</em> for these websites. However, in addition to the production <em>environment</em>, large projects will have additional <em>environments</em>, where code is developed and tested before being deployed to the production <em>environment</em>. The combination of these <em>environments</em> make up the <em>project</em>. Drupal 8 is built to enable developers to migrate functionality for a project between <em>environments</em>, using the Configuration API.</p> <h2>Components of a Drupal environment</h2> <p>What makes up an environment of a Drupal project? In other words, when making a 'copy' of a Drupal site, what are the elements that need to be migrated/copied to create that copy? A Drupal environment consists of the following elements:</p> <ol><li><strong>Codebase</strong>: At the core of any Drupal system is codebase that makes up the 'engine' that runs Drupal. The codebase of any Drupal site consists of Drupal core, modules (both contributed and custom), and themes (again, both contributed and custom). The codebase is a series of files, and these files provide the functionality of a Drupal site.</li> <li><strong>Configuration</strong>: Configuration is the collection of settings that define how the project will implement the functionality provided by the codebase. The codebase provides the abstract functionality to create things like content types, fields and taxonomies. Developers configure a Drupal site through the admin interface, to create the actual content types, fields, and taxonomies used in the project, that will allow end-users to do things like create pages or make comments. As developers build out the functionality of the project through the admin interface, the settings they choose are saved as the configuration for that environment. Configuration is environment-specific, and the Configuration API provides a means of migrating configuration between environments.</li> <li><strong>Content</strong>: Content is the data on a site that is specific to the given environment. Content may be created by end-users, or by content admins. While content, like configuration, sometimes needs to be migrated between environments, content should be considered environment-specific. The Migrate API provides a means of migrating content between environments.</li> <li><strong>Uploaded files</strong>: Most Drupal projects will have files uploaded as content. These are not part of the codebase that provides the functionality of the system, rather they are files that are to be provided to end users. Examples are images, audio/video files and PDFs.</li> <li><strong>Third party technologies</strong>: Drupal 8 is a hub framework, built to integrate with 3rd party libraries, APIs, softwares, and other technologies. For example, a project may use a 3rd party CSS/JS framework, or swap out the default database caching backend to a Redis caching backend, which provides significant performance improvements. While these third party technologies are not part of Drupal, often the technologies will be required in each environment for the project to run.</li> </ol><p>The above five elements together make up a Drupal environment. The environment is a fully-functioning Drupal copy (instance), with its own configuration and content. Each environment will have its own domain (URL) at which the environment can be accessed.</p> <h3>Types of environments</h3> <p>Typical projects these days will have a minimum of three environments:</p> <ol><li><em>Production</em> environment: The main live environment that end users will use.</li> <li><em>Staging</em> environments: One or more environments on which site owners can test new functionality or bug fixes before they are deployed to the production environment.</li> <li><em>Development</em> environments, where new functionality can be developed in isolation. Development environments may be publicly accessible, or may only exist on a developers computer, not accessible to the outside internet.</li> </ol><h2>Content and configuration - both stored in the database</h2> <p>A major issue that existed with all versions of Drupal core up to and including Drupal 7, was that content and configuration were both stored in the database, and there was no way to separate them. This made it difficult to manage configuration changes, such as adding a field to a content type, between multiple environments. There was no way to export the configuration for that single field, so to ensure consistency full databases were migrated. The problem is that database migrations are an all-or-nothing venture; when migrating from one environment to another, the entire database of the source environment overwrote the entire database on the target environment, meaning all content and configuration was overridden on the target environment.</p> <p>This opens up a problem though. Imagine this scenario:</p> <ol><li>A developer copies the production database to their local computer.</li> <li>Someone creates new content on the production environment, that becomes part of the database.</li> <li>The developer adds a new field on their local environment.</li> <li>The developer migrates their local database to the production environment, overwriting all configuration and content on the production environment.</li> </ol><p>With the above scenario, the new field created on the developer's local environment is now on the production environment, however the content created in step 2 is overwritten and lost. As such, the above process is untenable. Databases cannot be migrated 'up' (to production) after a site has gone live. They can only be migrated 'down' (from production).</p> <p>To overcome this problem, a common way to make changes to configuration before Drupal 8 was to make configuration changes on the production environment, then copy the production environment's database down to the other environments. For example, if a developer was writing code that depended on a new field, rather than create the field on their local development environment, they would create the field on the production environment, then copy the production environment database down to their local development environment. Now the field exists on both the production environment as well as their development environment. This solution works, but is akin to killing a mosquito with a sledgehammer - it's overkill. The developer did not need the whole database, only the configuration for that single field. And any changes they had already made in their own environment, such as content for testing, is now wiped out with data from the production environment.</p> <p>While this process worked (and still does continue to work for many projects), it was not ideal. It led to issues where database migrations had to be coordinated with clients and other developers to ensure that it was safe to wipe out content or configuration someone else may still be working with.</p> <h2>The Configuration API</h2> <p>The Drupal 8 Configuration API was created to overcome the above issue, so that developers could migrate configuration - such as Field API fields, content types, and taxonomies - between environments, without having to overwrite the entire database. It decoupled configuration from content. To understand how it makes things better, l<span>et's again review the components of a Drupal site, and the storage mechanisms behind them:</span></p> <ul><li>Codebase (files)</li> <li>Configuration (database)</li> <li>Content (database)</li> <li>Uploaded files (files)</li> </ul><p>With the Configuration API, configuration can be exported into files, and these files can be migrated between environments and imported into the new environment. This means that we can now migrate individual items of configuration without having to overwrite the entire database. The Configuration API decouples configuration from content. In the next parts of this series, we will explore how the API work</p> <p>Note 1: the Drupal 7 <a href="https://www.drupal.org/project/features">Features module</a> essentially does the same thing, and though the methodology is different, many of the concepts in the following articles will be relevant to that module as well.</p> <p>Note 2: the Drupal 8 <a href="https://www.drupal.org/docs/8/api/migrate-api/migrate-api-overview">Migrate API</a> was developed in parallel to the Configuration API, allowing for content also to be migrated, again without overwriting the entire database.</p> <h2>Summary</h2> <p>In this article, we looked at an overview of the Configuration API to understand what it is, and why it was created. After reading this article, you should have an understanding of what a Drupal environment is, and what it consists of. You should understand that in Drupal 8, configuration is stored in the database, but can be exported to files, which can then be used to migrate content between environments. In <a href="https://www.morpht.com/blog/drupal-8-configuration-part-2-how-api-works">Part 2 - How the API works</a>, we will take a closer, more technical look at how the API works, and in <a href="/blog/drupal-8-configuration-part-3-using-api">Part 3 - Using the API</a>, we will discuss how to use the API, and some contributed modules that extend its functionality.</p> Fri, 10 May 2019 12:18:26 +1000 Jay Friendly https://www.morpht.com/blog/drupal-8-configuration-part-1-configuration-api Drupal and Composer: Composer for Drupal Developers https://www.morpht.com/blog/drupal-and-composer-part-4-composer-drupal-developers <p class="introduction">As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those without previous experience working with command line, and can still be a confusing experience for those who do.</p> <p>This is the fourth post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:</p> <ul><li>Part 1: <a href="/blog/drupal-and-composer-part-1-understanding-composer">Understanding Composer</a></li> <li>Part 2: <a href="/blog/drupal-and-composer-managing-drupal-8-site-composer">Managing a Drupal 8 site with Composer</a></li> <li>Part 3: <a href="/blog/drupal-and-composer-converting-existing-drupal-8-site-composer">Converting Management of an Existing Drupal 8 Site to Composer</a></li> <li>Part 4: Composer for Drupal Developers</li> </ul><p>If you have not yet read <a href="https://www.morpht.com/blog/drupal-and-composer-part-1-understanding-composer">part 1</a> and <a href="https://www.morpht.com/blog/drupal-and-composer-managing-drupal-8-site-composer">part 2</a>, then before reading through this post, it is probably best to ensure you understand the concepts outlined in the summaries of those articles before moving forward.</p> <h2>Composer for Drupal Developers</h2> <p>The final section in this series addresses how Drupal developers can integrate profiles, modules and themes with Composer, to manage 3rd party libraries and/or other Drupal modules used by their code. To understand when and why a developer would wish to do this, let's have a look at a use case.</p> <h3>Use Case 1: Instagram Integration</h3> <p>Imagine a developer building a module to integrate with Instagram, pulling the most recent images from an Instagram account and displaying them on the Drupal site. Instagram provides an API for retrieving this information, but before Instagram provides this data, the Drupal site is required to authenticate itself to Instagram, using the OAuth2 protocol. After authenticating, various API calls can be made to retrieve data from the API using the the OAuth2 authentication token.</p> <p>Up to Drupal 7, the Drupal way to do this would be to write a custom OAuth 2 integration (or use the <a href="https://www.drupal.org/project/oauth2_client">OAuth2 Client module</a>), then write custom functions that made the API calls to Instagram, retrieving the data.</p> <p>Drupal 8 however, using Composer, acts like a hub, allowing for the inclusion of 3rd party libraries. This saves developers from having to re-create code within Drupal (aka <em>rebuilding the wheel</em>). For example, Instagram integration can be handled with the <a href="https://github.com/cosenary/Instagram-PHP-API">Instagram-PHP-API library</a>. This library provides OAuth2 integration specific to Instagram, as well as various PHP wrappers that interact with Instagram's API to handle the retrieval of data from the API. Using Composer, our developer can integrate this 3rd party library into Drupal, and use it in their Drupal code. This saves our developer the time and effort of creating and testing the code that integrates with Instagram, as these tasks are handled by the library maintainers. If the library is well maintained, it will be updated in line with updates to Instagram's API, saving developer time (and therefore client money) in maintenance as well. And finally, as the library is open source, it will have the benefit of having eyes on it by PHP developers from various walks, rather than just Drupal developers, making for a stronger, more secure library.</p> <h3>Use Case 2: Drupal Module Dependency</h3> <p>Imagine that it is a requirement that the images retrieved from Instagram be shown in a popup (aka modal or lightbox). This can be handled using the Drupal <a href="https://www.drupal.org/project/colorbox">Colorbox module</a>. As such, the Colorbox module will be set as a dependency in the Instagram module's <code>.info.yml</code> file. When a site builder managing their Drupal project with Composer downloads/manages our developer's Instagram module with Composer, then tries to enable the Instagram module in Drupal, they will get an error from Drupal that the Colorbox module is missing. The problem here is that Drupal is being told the Colorbox module is a dependency, but we are not managing that dependency with Composer, and therefore the code has not been downloaded and does not exist.</p> <p>At this point, it is easy to think that site builders could just add the Colorbox module to Composer with a <code>require</code> command. This would work, and after doing so, they would be able to enable the Instagram module, since the Colorbox module now exists. This opens up a problem however; imagine the site builder later decides to remove our developer's Instagram module. They disable/uninstall it in Drupal, then use <code>composer remove</code> to remove it. Everything looks good - the module has been uninstalled from Drupal, and the code has been removed from the codebase. However, there is one more step to return the system to its original state; disabling/uninstalling the now unnecessary Colorbox module from Drupal and removing the code using Composer. The necessity of this additional step opens up situations where the module will be left on the system due to forgetfulness, a lack of documentation, or changing site builders, creating site bloat due to unnecessary code existing and being executed.</p> <p>The solution to this is to also manage Drupal profile/module/theme dependencies with Composer. In our use case, our developer will list the Colorbox module as a dependency of the Instagram module not just in the Instagram module's <code>.info.yml</code> file, but also as a Composer dependency in <code>composer.json</code>. This way, when the Instagram module is added to a project using Composer, Composer will also download and manage the Colorbox module. And when the site builder removes the Instagram module, Composer will remove the Colorbox module as well (if no other libraries list it as a dependency).</p> <h2>Integrating Drupal Code with Composer</h2> <h3>Step 1: Create a composer.json file</h3> <p>Integrating with Composer requires that a <code>composer.json</code> file be created in the root of the profile/module/theme (from here on out referred to as the <em>Drupal library</em>) folder. This is explained in detail on the Drupal.org documentation page <a href="https://www.drupal.org/docs/8/creating-custom-modules/add-a-composerjson-file">Add a composer.json file</a>. Once this file has been added to the Drupal Library, it should be validated. This can be done by running <code>composer validate</code> on the command line in the same directory that the <code>composer.json</code> file lives in.</p> <p>Note that Drupal.org automatically adds a <code>composer.json</code> file to projects that do not have one. This is so that Composer is able to manage all projects on Drupal.org, not just projects to which maintainers have explicitly added a <code>composer.json</code> file.</p> <h3>Step 2: Declare your dependencies to Composer</h3> <p>All library dependencies, whether 3rd party libraries, or contributed Drupal libraries (modules etc), need to be declared as dependencies to Composer. This is done by calling <code>composer require [LIBRARY NAME]</code>, from within the folder of the Drupal library. This means that if you are adding dependencies for a <em>module</em>, you will navigate to the module folder, and add your dependencies. If you are adding dependencies for a <em>theme</em>, navigate to that folder, and add the dependencies there. The key point here is to not add your dependencies from the wrong folder, as they will be be added to the <code>composer.json</code> file of the wrong package.</p> <h4>Adding Remote Library Dependencies</h4> <p>In the use case example for this article, the <a href="https://github.com/cosenary/Instagram-PHP-API">Instagram-PHP-API library</a> was suggested for integration with Instagram. This library has the Composer key <code>cosenary/instagram</code>. To set this library as a dependency of a module, navigate to the root of the module, and run the following:</p> <p><code>composer require cosenary/instagram</code></p> <p>Running this code results in the following:</p> <ol><li>The <em>cosenary/instagram</em> library is added as a dependency to the <code>composer.json</code> file, so that Composer knows the package is managed by Composer. The <code>composer.json</code> file will contain something like the following:<br />   <p><code>"require": {<br />   ...<br />   "cosenary/instagram": "^2.3",<br />   ...<br /> } </code><br /> Note that composer.json is committed to Git.</p> </li> <li> <p>The [MODULE ROOT]/vendor folder is created, and the Instagram-PHP-API library is downloaded into this folder. Composer by default creates the <code>vendor</code> folder in the same directory as the <code>composer.json</code> file. </p> <p> Note that once the Drupal library has been pushed to a repository where it can be managed by Composer, installing the Drupal library with Composer will install the Instagram-PHP-API library to the project's vendor folder. As such, it's a good idea to delete the vendor folder in the module after the new <code>composer.json</code> and <code>composer.lock</code> files have been committed to the remote repository, before requiring the module with Composer.</p> <p> As a best practice, it is a good idea to create a <code>.gitignore</code> file in the module root, and include the vendor directory to ensure that it is never accidentally committed. </p> </li> <li> <p>The <code>composer.lock</code> file is created, locking the installed version of the Instagram-PHP-API library to the Instagram module. This ensures that users of the Instagram library are working with a compatible version of the Instagram-PHP-API library. Note that this file is also committed to Git.</p> </li> <li> <p>Any dependencies of the Instagram-PHP-API library, declared in its own <code>composer.json</code> file, are downloaded.</p> </li> <li> <p>All libraries and dependencies are checked for version conflicts.</p> </li> </ol><h4>Adding Drupal Library Dependencies</h4> <p>In the use case example for the article, the Drupal Colorbox module (aka library) was mentioned to be a dependency of the Instagram module, and therefore is to be managed with Composer. Adding Drupal modules as dependencies requires an additional step, due to their being hosted on a non-default repository. To add a Drupal library as a dependency:</p> <ol><li>Edit the <code>composer.json</code> file and add the Drupal repository to it, so that Composer knows where to look for the Drupal libraries, including the Colorbox module: <p> <code>"repositories": [<br />   {<br />     "type": "composer",<br />     "url": "https://packages.drupal.org/8"<br />   }<br /> ]</code></p> <p> Now Composer knows where to look for the Colorbox module when the Instagram module is installed.  If developers try to declare Colorbox module as a dependency before this step, they will get an error that Composer cannot find the <code>drupal/colorbox</code> library.<br />  </p></li> <li>Add the module with composer require: <p> <code>composer require drupal/colorbox</code></p> <p> Note that this will download the Colorbox module to <code>[MODULE ROOT]/vendor/drupal/colorbox</code>. This is NOT a good thing, for while Drupal will be able to find the module in this location, it is not the standard location, and if another copy of the module ends up in other directories that Drupal scans when looking for modules this will create hard to debug issues. So either delete the Colorbox module in the vendor folder right away, or immediately move it to the location where the rest of your contributed modules are located, so it's very clear where it is at.<br />  </p></li> </ol><p>After this, both the <code>composer.json</code> file and the the <code>composer.lock</code> file should be committed to Git. The module now has its dependencies declared on both the Instagram-PHP-API library and the Drupal Colorbox module. When site builders install the module to their Drupal project using Composer, the Instagram library will be downloaded to the project (site) <code>vendor</code> folder, and the Drupal Colorbox module will be installed to the <code>web/modules/contrib</code> folder.</p> <h2>Developing With Composer-Managed Drupal Libraries</h2> <p>Now that the module has been integrated with Composer, Composer can be used to manage the module during development. Let's imagine that the Instagram module being developed has the key <code>drupal/insta</code>, and is on Version 8.x-1.x. Downloading the -dev version of the module can be done by appending <code>:1.x-dev</code> when requiring the module:</p> <p><code>composer require drupal/insta:1.x-dev</code></p> <p>This will download the 8.x-1.x-dev version of module to the <code>web/modules/contrib</code> folder, where it can be worked with. A <span>.git</span> folder will be included in the module root, so that changes can be committed to Git. Sounds great, but when our developer tries to push their commits, they will get an error, as the remote URL of the repository is not the repository for maintainers of the module and does not allow commits. As such, we need to change the Git repository's remote URL to the URL of the repository for maintainers, allowing for code to be pushed, and releases to be made.</p> <p>To find the correct Git repository URL, first go to the module's download page on Drupal.org. Make sure you are logged in as a maintainer of the module. If you are a maintainer and have the correct permissions, you will see a tab called <em>version control</em>. Click this tab. Next, copy the Git URL on the page. It will look something like this:</p> <p><code>git@git.drupal.org:project/</code>[MODULE NAME]<code>.git</code></p> <p>Next, navigate to your module folder, and run the following commands:</p> <p><code>git remote rm origin<br /> git remote add git@git.drupal.org:project/</code>[MODULE NAME]<code>.git</code></p> <p>The first line of this code removes the incorrect remote Git URL for non-maintainers, and the second line adds the correct Git remote URL for maintainers. After this, you will be able to push changes, including tags (releases).</p> <p>You can also then easily switch between your development and release versions of the module by alternating the following:</p> <p><code>composer require drupal/insta</code></p> <p><code>composer require drupal/insta:1.x-dev</code></p> <p>Note however that when doing any commits, you will need to switch up the remote URL each time you switch to the dev version of the module.</p> <h2 class="display-xxl">Summary</h2> <p>In this final part of the series on using Composer with Drupal, we have looked at how Drupal developers can integrate their custom code with Composer, to ensure that dependencies of the module are correctly managed. We have also looked at how to develop modules that are being managed with Composer. Using these techniques will allow site builders to keep a clean codebase that manages library conflicts.</p> <p>And this concludes my four-part series on Drupal and Composer. I hope you have enjoyed it, and that it shows just how powerful Composer is, and how it can be effectively used to manage Drupal 8 sites. <strong>Happy Composing and Happy Drupaling!</strong></p> Tue, 22 Jan 2019 04:00:00 +1100 Jay Friendly https://www.morpht.com/blog/drupal-and-composer-part-4-composer-drupal-developers Drupal and Composer: Converting an existing Drupal 8 site to Composer https://www.morpht.com/blog/drupal-and-composer-part-3-converting-management-existing-drupal-8-site-composer <p class="introduction">As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those without previous experience working with command line, and can still be a confusing experience for those who do.</p> <p>This is the third post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:</p> <ul><li>Part 1: <a href="/blog/drupal-and-composer-part-1-understanding-composer">Understanding Composer</a></li> <li>Part 2: <a href="/blog/drupal-and-composer-managing-drupal-8-site-composer">Managing a Drupal 8 site with Composer</a></li> <li>Part 3: Converting Management of an Existing Drupal 8 Site to Composer</li> <li>Part 4: <a href="https://www.morpht.com/blog/drupal-and-composer-composer-drupal-developers">Composer for Drupal Developers</a></li> </ul><p>If you have not yet read <a href="https://www.morpht.com/blog/drupal-and-composer-part-1-understanding-composer">part 1</a> and <a href="https://www.morpht.com/blog/drupal-and-composer-managing-drupal-8-site-composer">part 2</a>, then before reading through this post, it is probably best to ensure you understand the concepts outlined in the summaries of those articles.</p> <h2>Switching Management of your Drupal site to Composer</h2> <p>So you’ve worked your way through parts one and two of this series, and you now understand what Composer is, how it can be used to work with Drupal 8, and how to start a new Drupal 8 project using Composer. But, you started your current project without using Composer, and want to switch to managing your project using Composer. Where do you start? This article will discusses a few strategies behind converting your existing system to Drupal. Fortunately some automated tools exist for converting existing sites to Composer, and in the situation that neither of these tools work, an overview is provided on how to manually convert an existing site </p> <p>But, before moving on to any of these methods...</p> <p><strong>Take a backup!</strong> As this process will be destructive to your system, make sure you take a backup of your file system, and take a backup of your database. Then go and check to ensure that you have a full backup of the file system, and a full back up of the database.</p> <p>If you skip this step, or do not do it properly, you may end up with an entirely broken system, so don’t skip this step.</p> <h2>Method 1: Composerize (Composer plugin)</h2> <p><a href="https://github.com/grasmash/composerize-drupal">Composerize Drupal</a> is a Composer plugin that has been built to convert existing Drupal installations to use Composer. Instructions are on the download page. If this method doesn't work, you can try the Drupal Composerize module:</p> <h2>Method 2: Composerize (Drupal module)</h2> <p>The <a href="https://www.drupal.org/project/composerize">Composerize module</a> is a Drupal module that is built to convert existing Drupal installations to use Composer. At the time of writing, this module has the following disclaimer on the page:</p> <blockquote><p>This module is still in development. It supports very basic Drupal 8 setups, but there are many necessary features it still lacks (e.g., support for patches, JavaScript libraries, distributions, etc). We're working on all that stuff, but <strong>this module is definitely not ready for prime time</strong>.</p> </blockquote> <p>If this method doesn't work, you'll likely have to manually convert your site.</p> <h2>Method 3: Manual method</h2> <p>If the above steps fail, your last option is to convert your installation to using the <a href="https://github.com/drupal-composer/drupal-project">Drupal Composer Template</a> manually. N<span>ote that pretty much every system will be different, so these instructions are an overview of the end goal, rather than a complete set of steps that will convert your system. There is a good chance you’ll run into issues along the way that are not covered here, so make sure you took that backup!</span></p> <p>Converting a system to the template requires achieving the following goals:</p> <ul><li>Setting up the file structure of the system to match the Drupal Composer Template</li> <li>Delete old Composer files</li> <li>Add the Template file system to your Drupal system</li> <li>Set up the configuration directory and the private files directory</li> <li>Set up your profiles, modules and themes to be managed by Composer</li> </ul><h3>Step 1: Convert your file structure to match the Drupal Composer Template file structure</h3> <p>The Drupal Composer Template is set up to manage the directory above the webroot, as well as the webroot. The webroot is located in the [PROJECT ROOT]/web folder. Therefore you will need this structure:</p> <ul><li><code>/</code> - Project root. Contains various scaffolding files such as composer.json and composer.lock, as well as the configuration export folder, and the webroot folder <ul><li><code>/web</code> - The webroot. Contains all Drupal core, profile, module and theme files.</li> </ul></li> </ul><p>You'll need to set up your Drupal installation to match this structure, and make sure that the server is set up so that the web root is at [PROJECT ROOT]/web.</p> <p><strong>Note</strong>: Some servers require the webroot to be in a directory with a specific name, such as <code>public_html</code>, or <code>www</code>. In this case, you can try setting up symlinks from the required directory name to the <code>/web</code> directory, so that, for example, <code>[PROJECT ROOT]/public_html</code> is a symlink pointing at <code>[PROJECT ROOT]/web</code>. Your Drupal files will then reside in the <code>/web</code> directory (satisfying the Drupal template), and also be accessible at <code>/public_html</code> (satisfying the server requirements). If this does not work, another option would be to edit the composer.json file (which is added in step 3) and change any paths that point at the <code>/web</code> directory, to point at the directory name you are actually using.</p> <h3>Step 2: Delete old Composer files</h3> <p>I'll say it again, because it needs to be said, make sure you took that backup in step 0!</p> <p>If any of the following files or folders exist in your installation, delete them. Note however that you may want to save the composer.json file for reference if you've manually added any libraries into your existing installation using Composer.</p> <ul><li><code>[PROJECT ROOT]/vendor</code> directory</li> <li><code>[PROJECT ROOT]/composer.json</code> file</li> <li><code>[PROJECT ROOT]/composer.lock</code> file</li> <li><code>[PROJECT ROOT]/web/vendor</code> directory</li> <li><code>[PROJECT ROOT]/web/composer.json</code> file</li> <li><code>[PROJECT ROOT]/web/composer.lock</code> file.</li> </ul><h3>Step 3: Add the Template file system to your Drupal system</h3> <p>Go <a href="https://github.com/drupal-composer/drupal-project/archive/8.x.zip">here</a>, click 'clone or download' and download the .zip file (note - or clone the Git repository if you prefer)</p> <ol><li>Save/move the zip file into the project root folder (the directory above the 'web' folder you created above). You can then unpack it using the following command. Before this step, the file <code>/composer.json</code> should not exist. After this step, if you've done it correctly, this file will exist.<br /><code>tar -zxvf [FILE] --strip-components=1</code></li> <li>Run the following command from the project root folder. This command will Install the Composer dependencies as well as create the <code>/vendor</code> directory.<br /><code>composer install</code></li> <li>Run the following to ensure your database is up to date, and caches are cleared.<br /><code>drush updb; drush cr;</code></li> </ol><h3>Step 4: Set up the configuration directory and the private files directory (Optional)</h3> <p>This next step is optional, however it will make for a more secure system. First, the following directories need to be created if they don't already exist:</p> <ul><li><code>[PROJECT ROOT]/config/sync</code></li> <li><code>[PROJECT_ROOT]/private</code></li> </ul><p>The first folder is where exports of the Drupal 8 configuration system will be exported to. The second folder is the private files folder. Creating both of these directories as siblings to the webroot adds security, as the files are not in a web-accessible location. The next thing to do is tell the system of the location of these files. This is done by declaring the folder paths in settings.php. You can do this by adding the following two lines to the bottom of settings.php:</p> <p><code>$config_directories['sync'] = '../config/sync';<br /> $settings['file_private_path'] = '../private';</code></p> <p>After this, clear the registry (<code>drush cr;</code>). You can confirm that the configuration directory was properly set by running <code>drush cex sync</code>, and then checking that there are .yml files in the <code>[PROJECT ROOT]/config/sync</code> directory. You can confirm that the private files folder was properly set by going to Admin -&gt; Configuration -&gt; Media -&gt; File System, and confirming that the private files directory is listed as <code>../private</code>.</p> <h3>Step 5: Set up your profiles, modules and themes to be managed by Composer</h3> <p>The final step is to set up Composer to manage Drupal profiles, modules and themes to be managed by Composer. The Drupal Composer Template tracks Core by default, but needs to be informed of the rest of your code. Note that if you do not want to use the most recent version of these profiles/modules/themes, you will need to alter the commands below to set the version you want to install.</p> <p>Drupal profiles, modules and themes can be installed with the following command:</p> <p><code>composer require drupal/[PACKAGE NAME]</code></p> <p>For example, if you were using the Administration Toolbar module (admin_toolbar), you would run:</p> <p><code>composer require drupal/admin_toolbar</code></p> <p>After you have done this, ensure you are up to date with a DB update and cache clear:</p> <p><code>drush updb; drush cr;</code></p> <p>At this point, your system should be converted to the Drupal Composer Template, with contributed code being managed by Composer.</p> <h2>Summary</h2> <p>This article looks at converting exiting Drupal 8 sites to being managed by the <a href="https://github.com/drupal-composer/drupal-project">Drupal Composer Template</a>. Doing this can potentially be automated using the Composerize Composer plugin or the Composerize Drupal module. In situations where this does not work, the manual directions in this article can be used as an alternative.</p> <p>In the next and final part of this series, we'll look at how Drupal developers can integrate 3rd party libraries into their custom Drupal profiles, modules and themes.</p> Thu, 03 Jan 2019 17:11:52 +1100 Jay Friendly https://www.morpht.com/blog/drupal-and-composer-part-3-converting-management-existing-drupal-8-site-composer Drupal and Composer: Managing a Drupal 8 site with Composer https://www.morpht.com/blog/drupal-and-composer-part-2-managing-drupal-8-site-composer <p class="lead">As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those who don't have previous experience working with the command line, and can still be a confusing experience for those who do. This is the second post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:</p> <ul><li>Part 1: <a href="/blog/drupal-and-composer-part-1-understanding-composer">Understanding Composer</a></li> <li>Part 2: Managing a Drupal 8 site with Composer</li> <li>Part 3: <a href="/blog/drupal-and-composer-converting-existing-drupal-8-site-composer">Converting Management of an Existing Drupal 8 Site to Composer</a></li> <li>Part 4: <a href="https://www.morpht.com/blog/drupal-and-composer-composer-drupal-developers">Composer for Drupal Developers</a></li> </ul><p>This article will be difficult to understand without first understanding the concepts explained in <a href="/blog/drupal-and-composer-part-1-understanding-composer">part 1</a>, so If you have not read it, it would probably be worth your while to ensure you understand the concepts outlined in the summary of that article, before proceeding with this one.</p> <h2>Beginning a New Project</h2> <p>Fortunately a lot of work has been put into creating a Composer base (called a template) for Drupal projects. This Drupal Composer template can be found on Github at: <a href="https://github.com/drupal-composer/drupal-project">https://github.com/drupal-composer/drupal-project</a>. Instructions on how to use the template can be found there, and the same instructions are found in the README.md file that comes with the project when installed.</p> <p>Starting a new project with the Drupal Composer template can be done with the following line of code:</p> <p><code>composer create-project drupal-composer/drupal-project:8.x-dev some-dir --stability dev --no-interaction</code></p> <p>This command does a number of things, many of which will be addressed below.</p> <h2>1) A new Composer project is created</h2> <p>The first thing the installation does is to create the directory specified as <em>some-dir</em> in the command, and initializes a Composer project in that directory. Change this to the appropriate name for your project. This is now your new project. The project contains the <em>composer.json</em> and <em>composer.lock</em> files that will be used to manage the code base of the project.</p> <div class="well">Note: The command provided lists the --<em>stability</em> as <em>dev</em>. This can be confusing as developers may think this means they are getting the -dev version of Drupal core. Don't worry, the command given above will always install the current full release of Drupal core, whatever it is at the time when the command is run.</div> <h2>2) Library dependencies (as well as their dependencies) are installed</h2> <p>The Composer template has a number of dependencies included by default, some of which we will take a look at here. These libraries are set by default as requirements in composer.json, and therefore are included when running the install command given earlier.</p> <ul><li><a href="https://www.drush.org/">Drush</a>: Drush is cool. If you don’t know what it is, it’s worth some Google-fu. Anything to be written about Drush has already been written somewhere else, so check it out - it will be worth your while!</li> <li><a href="https://drupalconsole.com/">Drupal Console</a>: Drupal Console is also really cool. See comments on Drush.</li> <li><a href="https://github.com/cweagans/composer-patches">Composer Patches</a>: This one is very cool. This library in and of itself is worth using Composer to manage Drupal projects in my eyes. Even if Composer had no other benefits, this one would be great. First, an explanation of a patch is necessary. A patch is kind of like a band-aid that can be applied to code. Patches allow developers to submit changes, be they bug fixes or new functionality, to the library maintainer. The library maintainer may or may not add the patch to the source code, but in the meantime, other developers can apply the patch to their own systems, both to test if it works, as well as use it if it does. However, when the library the patch has been applied to is updated to a newer version, the patches have to be re-applied. What Composer Patches does is allow developers to track patches applied to the project, and have them applied automatically during the update process. This ensures that bugs don't arise from forgetting to re-apply patches after the update. Patches are tracked by adding them to composer.json. Here is an example: <p> <code>"extra": {<br />   "patches": {<br />     "drupal/core”: {<br />       “Patch description”: "https://www.drupal.org/files/issues/someissue-1543858-30.patch"<br />     }<br />   }<br /> }</code><br />  </p></li> <li> <p>With the above code, the next time <code>composer update drupal/core</code> is run, Composer will attempt to apply the patch found at <a href="https://www.drupal.org/files/issues/someissue-1543858-30.patch">https://www.drupal.org/files/issues/someissue-1543858-30.patch</a> to Drupal core. Note that the description of the patch, given above as "Patch description", is arbitrary, and should be something descriptive. If there is an issue for the patch, a link to the issue is good to add to the description, so developers can quickly look into the status of the patch, see if any updated patches have been released, and check if the patch has been incorporated into the library, rendering the patch unnecessary.</p> </li> <li>And Others: There are many other libraries that are included as part of the Drupal Composer template, but the truth is that I haven’t looked into them. Also note that Drupal core alone has multiple dependencies which have their own dependencies. At the time of writing, 123 libraries are installed into the project with the install command.</li> </ul><p><strong><em>But wait - I don’t use [fill in library here]</em></strong></p> <p>The Composer template is just that - a template. Some people don’t use Drush, some don’t use Drupal console. If these are not needed, they can be removed from a project in the same manner as any Composer library. Example:</p> <p><code>composer remove drush/drush</code></p> <p>The above command will remove Drush from the code managed by the Composer template.</p> <h2>3) The system folder architecture is created</h2> <p>The file system created by the Composer template deserves a close look.</p> <ul><li><strong>/web:</strong> The first thing to notice is that Drupal is installed into the /web directory in the root of the project. This means that the root of the project created with the Composer template is one level above the webroot. When configuring the server, the domain for the server will need to point at the /web directory.</li> <li><strong>/config/sync:</strong> The Composer template sets up the project to store Drupal 8 configuration .yml files in this folder. When running <code>drush cex sync</code> to export configuration, the entire site configuration will be exported to this folder. This is folder is best kept out of the webroot for security purposes.</li> <li><strong>/drush: </strong>This folder holds a few Drush specific items in it. If multiple environments exist for your project, Drush aliases can be set in /drush/sites/self.site.yml, allowing for interaction with your various environments from anywhere within the project.</li> <li><strong>/scripts:</strong> At the time of writing, this folder contains only a single file, /scripts/composer/ScriptHandler.php. This is a really cool file that contains code run by Composer during various processes. <p> The composer.json file in the Drupal Composer template contains the following:<br />  </p> <p><code>"scripts": {<br />   "drupal-scaffold": "DrupalComposer\\DrupalScaffold\\Plugin::scaffold",<br />   "pre-install-cmd": [<br />     "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"<br />   ],<br />   "pre-update-cmd": [<br />     "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"<br />   ],<br />   "post-install-cmd": [<br />     "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"<br />   ],<br />   "post-update-cmd": [<br />     "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"<br />   ]<br /> },</code></p> <p>The code above executes the stated code on pre-install, pre-update, post-install and post-update. Any time either <code>composer install</code> or <code>composer update</code> are executed on the system, the pre and post hook for that call are executed, calling the relevant functions above. Developers can create their own pre/post install and update hooks following the examples shown above.</p> </li> <li> <p><strong>/vendor: </strong>The first thing to note is that the location of this file differs from a vanilla Drupal 8 installation. When installing Drupal manually, the vendor folder is part of the webroot by default. This however could lead to security issues, which is why the Composer template installs it above the webroot. The vendor folder contains most of the libraries that Composer manages for your project. Drupal core, modules, themes and profiles however are saved to other locations (to be discussed in the next section). Everything else is saved to the /vendor folder.</p> </li> </ul><h2>4) Drupal File/Folder Installation Locations are set</h2> <p>As mentioned above, Drupal core is installed into the /web folder. The Composer template also sets up installation locations (directories) for Drupal libraries, modules, themes and profiles so that when Composer installs these, they are put in the appropriate Drupal folder locations. This is the code in composer.json that handles the installation locations:</p> <p><code>"extra": {<br />   "installer-paths": {<br />     "web/core": [<br />       "type:drupal-core"<br />     ],<br />     "web/libraries/{$name}": [<br />       "type:drupal-library"<br />     ],<br />     "web/modules/contrib/{$name}": [<br />       "type:drupal-module"<br />     ],<br />     "web/profiles/contrib/{$name}": [<br />       "type:drupal-profile"<br />     ],<br />     "web/themes/contrib/{$name}": [<br />       "type:drupal-theme"<br />     ],<br />     "drush/contrib/{$name}": [<br />       "type:drupal-drush"<br />     ]<br />   }<br /> }</code></p> <p>The most common library type that a Drupal developer will install will be Drupal modules. So let’s look at the line of code specific to the module installation location:</p> <p><code>"web/modules/contrib/{$name}": [<br />   "type:drupal-module"<br /> ],</code></p> <p>This line of code says that if the type of Composer library is drupal-module, then install it to the <code>/web/modules/contrib/[MODULE MACHINE NAME]</code> folder.</p> <p>But how does Composer know that the library being downloaded is type drupal-module? Well, the key to this is in how Composer libraries are managed in the first place. Throughout this article and the one that precedes it, we have repeatedly looked at the composer.json file that defines this project. Well, every Composer library contains a composer.json file, and every composer.json file that comes with packaged with Drupal modules contains a type declaration as follows:</p> <p><code>"type": "drupal-module",</code></p> <p>When Composer is installing libraries, it looks for a type declaration, and when it finds one, if there is a custom install location set in composer.json for that type, the library is installed to the declared folder. Drupal themes are of type drupal-theme, Drupal profiles are of type drupal-profile, and so on, and they are installed to the folders declared in composer.json.</p> <h2>Managing Custom Drupal Code with Composer</h2> <p>Custom code for a project can be managed with Composer as mentioned in <a href="https://www.morpht.com/blog/drupal-and-composer-part-1-understanding-composer">Part 1</a> of this series. Drupal convention generally separates contributed (3rd party) modules and custom modules into separate folders. To install custom code in the locations according to this convention, the following lines should be added to the installer-paths declaration in composer.json:</p> <p><code>"web/modules/custom/{$name}": [<br />   "type:drupal-custom-module"]<br /> ,<br /> "web/profiles/custom/{$name}": [<br />   "type:drupal-custom-profile"<br /> ],<br /> "web/themes/custom/{$name}": [<br />   "type:drupal-custom-theme"<br /> ],</code></p> <p>This code adds three additional library types for custom modules, custom profiles, and custom themes.</p> <p>Next, you’ll need to add a composer.json file to the custom module/theme/profile. Directions for this can be seen here: <a href="https://www.drupal.org/docs/8/creating-custom-modules/add-a-composerjson-file">https://www.drupal.org/docs/8/creating-custom-modules/add-a-composerjson-file</a>. Note that for the step named Define your module as a PHP package, you should set the type as <code>drupal-custom-[TYPE]</code>, where [TYPE] is one of: module, theme, or profile.</p> <p>Continuing on, make sure the composer.json file containing the type declaration has been pushed to the remote private repository.</p> <p>The last step step is to add your private repository to you project’s composer.json, so that when running <code>composer require my/privatelibrary</code>, Composer knows in which repository to look for the library. Declaring private repositories in composer.json is explained here: <a href="https://getcomposer.org/doc/05-repositories.md#using-private-repositories">https://getcomposer.org/doc/05-repositories.md#using-private-repositories</a>.</p> <p>With the above steps, when running <code>composer install my/library</code>, Composer will find the private repository declared in composer.json, search that repository for my/library, and download it. The composer.json file in my/library tells Composer that it’s of type <code>drupal-custom-[TYPE]</code>, so Drupal will install it into the directory specified for Drupal custom [TYPE].</p> <p>If using Git for version control on your system, you'll probably want to alter the .gitignore file in the Composer project root to ignore the custom folder locations. If you have created a custom module, and will be managing all custom modules with Composer and private repositories, you should probably add the /web/modules/custom folder to .gitignore. If you will be managing some custom modules with Git and not Composer, then you should probably add the custom module you have created to .gitignore as /web/modules/custom/[MODULE NAME].</p> <h2>Managing site settings: settings.php and settings.local.php</h2> <p>This section isn’t actually directly related to Composer and Drupal, but it’s a good step for setting up a project, and we can use the methodology to work with Composer template and Git.</p> <p>Each Drupal installation depends on the file settings.php. This file is loaded as part of Drupal’s bootstrap process on every page load. Site-specific settings are added into this file, such as database connection information.</p> <p>Towards the bottom of settings.php, the following lines can be found:</p> <p><code># if (file_exists($app_root . '/' . $site_path . '/settings.local.php')) {<br /> # include $app_root . '/' . $site_path . '/settings.local.php';<br /> # }</code></p> <p>These lines are commented out by the # symbol at the start of each line. This means that the code is not executed. If these lines are uncommented, by removing the hash symbol at the start of each line, this code is executed. The code looks for a file named settings.local.php, and if it exists, it includes that file. This means any settings in settings.local.php become part of the bootstrap process and are available to Drupal. After uncommenting the code, it will look like this:</p> <p><code>if (file_exists($app_root . '/' . $site_path . '/settings.local.php')) {<br />   include $app_root . '/' . $site_path . '/settings.local.php';<br /> }</code></p> <p>Why do this? This allows settings to be split into two types: settings that will be the same across all environments (eg. production, staging, local installations etc), and local settings that are only relevant to the current Drupal environment in which this file exists. This is done by committing settings.php to Git so it is shared amongst every environment (note - settings.local.php is NOT committed to Git, as it should not be shared). For example, the majority of Drupal installations have a separate database for each environment, meaning that database connection details will be specific to each environment. Therefore, database settings would be put into settings.local.php, as they are not shared between environments. The connection details to a remote API however may be the same regardless of environment, and therefore would be put into settings.php. This ensures any developer working on the system has access to the API details.</p> <p>After splitting up the settings this way, settings.php is committed to Git so it is tracked and shared between environments. </p> <p>The full process is as follows:</p> <ol><li>Install the Drupal Composer template as described earlier in this article</li> <li>Uncomment the code in settings.php as explained above</li> <li>Install Drupal as normal</li> <li>Run <code>git diff settings.php</code> to see what has been added to settings.php as part of the installation process. Anything that shows up should be added to settings.local.php, and removed from settings.php. This will definitely be the database connection, but could be other files as well.</li> <li>Edit .gitignore in the Composer project root, and remove this line: <p>/web/sites/*/settings.php</p> <p> You can now add settings for all environments to settings.php and settings specific to the local environment to settings.local.php.</p></li> </ol><h2>Setting Up the Private Files Directory</h2> <p>After setting up settings.php and settings.local.php as described above, you can now add the following to settings.php:</p> <p><code>$settings['file_private_path'] = ‘../private’;</code></p> <p>Next, create the folder <code>/private</code> in the root of your Composer project. Finally clear the Drupal cache, which will create the file /private/.htaccess. At this point you can now add settings.php and the /private folder to Git. Finally, edit .gitignore and add the following:</p> <p><code># Ignore private files</code></p> <p><code>/private/</code></p> <p class="display-md">This sets up the private file directories across all installations, saving developers having to set it up for each installation. Note that the .gitignore setting will ensure the contents of this folder are ignored by Git, as they should be.</p> <h2>What should I add to Git?</h2> <p>The Drupal Composer template project page states:</p> <blockquote><p>You should create a new git repository, and commit all files not excluded by the .gitignore file.</p> </blockquote> <p>In particular, you will need to ensure you commit composer.json and composer.lock any time composer changes are made.</p> <h2>Is It Safe to Use Composer on a Production Server?</h2> <p>The actual answer to this question is not one I have. I am not a server guy overall, and for that matter, I’m not even that much of a Composer expert. It may be that Composer is entirely safe on a production server, but personally my thoughts are that having a program that can write files to the server from remote servers would seem to open up a potential security risk, and therefore it’s likely better to NOT have Composer on a production server. This comment may lead you to question why I would have wasted the time to write so much on Composer if it’s better to not use it in the first place. But not so fast partner! It really depends on the system architecture and the server setup. Some servers have been set up so that as part of the deployment process, the codebase is built using Composer, but then set up as a read-only file system or a Docker container or some other process ensuring security. This however is a particularly complex server set up. Fortunately there is an alternative for developers who are not working with servers configured in this fashion, which we'll look at next.</p> <h2>Using Composer, without having Composer installed on the production server</h2> <p>There is an in-between solution that allows us to use Composer to manage our projects, even with multiple developers, while not having Composer installed on the production server. In this case, we can use a hybrid of a Composer managed project and the old style of using pure Git for deployment.</p> <p>First, we need to edit the .gitignore folder in the root of our Composer installation. In particular, we need to remove the following code:</p> <p><code># Ignore directories generated by Composer<br /> /drush/contrib/<br /> /vendor/<br /> /web/core/<br /> /web/modules/contrib/<br /> /web/themes/contrib/<br /> /web/profiles/contrib/<br /> /web/libraries/</code></p> <p>The above directories are all created by Composer and managed by Composer, which is why they were originally ignored by Git. However, we will not be managing our production server using Composer, and therefore we want to include these folders into the Git repository rather than ignoring them.</p> <p>After setting up your project, commit the folders listed above to Git. This ensures all the files that are managed by Composer will be part of Git. That way<code>composer install</code> never needs to be run on the production server, since any code that command would download will already be part of Git.</p> <p>What this means now is that any time a developer on the project add/updates code by running either <code>composer update</code> or <code>composer install</code>, they will need to commit not just the composer.json and composer.lock files, but also the added/updated source files that Composer manages, so that all code be available on the production server when checking out the code from Git.</p> <h2>Updating Drupal using Composer</h2> <p>In <a href="/blog/drupal-and-composer-part-1-understanding-composer">part one</a> of this series, I discussed library versions. I am not going to go deep into how the versioning works internally, but I’ll explain how updating works specific to Drupal core. At the time of writing, the current version of Drupal is 8.6.3. The dependency set in composer.json is for Drupal 8.6.*. The * at the end of this means that your project uses upon any version of Drupal 8.6, so when a minor version update comes out for 8.6, for example 8.6.4, Drupal core will be updated to Drupal 8.6.4 when <code>composer update drupal\core</code> is run.</p> <p>However, when Drupal 8.7 is released, it will not be automatically installed, since it does not fit the pattern 8.6.*. To upgrade to Drupal 8.7, the following command is run:</p> <p><code>composer update drupal/core:~8.7</code></p> <p>The <code>~/8.7</code> part of the above command tells Composer to use any version of Drupal 8.7. This means that in the future, when you run <code>composer update drupal/core</code>, minor releases of the 8.7 branch of Drupal core will be installed.</p> <h2>Summary</h2> <p>In this article, we have gone over how Composer is used with Drupal to make project deployment a little smoother, more stable, and consistent between environments. You should have an understanding of:</p> <ul><li>How to set up a new Drupal project using Composer</li> <li>How the following folders relate to the project: <ul><li>/config</li> <li>/drush</li> <li>/scripts</li> <li>/web</li> <li>/vendor</li> </ul></li> <li>How Composer adds Drupal modules and/or themes</li> <li>Drupal library dependencies</li> <li>Managing custom Drupal code with Composer</li> <li>Which items should be committed to Git</li> <li>How to use Composer to manage Drupal projects where the production server does not have Composer</li> <li>How to update Drupal using Composer</li> </ul><p>In the next post, coming soon, we'll look at how to convert an existing Drupal project to being managed by Composer.</p> Tue, 04 Dec 2018 10:04:28 +1100 Jay Friendly https://www.morpht.com/blog/drupal-and-composer-part-2-managing-drupal-8-site-composer Drupal and Composer: Part 1 — Understanding Composer https://www.morpht.com/blog/drupal-and-composer-part-1-understanding-composer <p class="lead">As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those without previous experience working with the command line, and can still be a confusing experience for those who do. This is the first post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:</p> <ul><li>Part 1: Understanding Composer</li> <li>Part 2: <a href="/blog/drupal-and-composer-managing-drupal-8-site-composer">Managing a Drupal 8 site with Composer</a></li> <li>Part 3: <a href="/blog/drupal-and-composer-part-3-converting-management-existing-drupal-8-site-composer">Converting Management of an Existing Drupal 8 Site to Composer</a></li> <li>Part 4: <a href="https://www.morpht.com/blog/drupal-and-composer-part-4-composer-drupal-developers">Composer for Drupal Developers</a></li> </ul><p>So without further ado, let’s get started.</p> <h2>Composer: What is it?</h2> <p>The Wikipedia page (<a href="https://en.wikipedia.org/wiki/Composer_(software)">https://en.wikipedia.org/wiki/Composer_(software)</a>) describes Composer as follows:</p> <blockquote><p class="lead"><strong>Composer</strong> is an <a href="https://en.wikipedia.org/wiki/Application-level_package_manager">application-level package manager</a> for the <a href="https://en.wikipedia.org/wiki/PHP">PHP</a> <a href="https://en.wikipedia.org/wiki/Programming_language">programming language</a> that provides a standard format for managing dependencies of PHP software and required <a href="https://en.wikipedia.org/wiki/Library_(computing)">libraries</a>.</p> </blockquote> <p>That’s an accurate description, though a little wordy. So let’s break it down a little further to understand what it means.</p> <p>Programmers like to use the term <strong>DRY</strong> - <strong>D</strong>on’t <strong>R</strong>epeat <strong>Y</strong>ourself. This means that whenever possible, code should be re-used, rather than re-written. Traditionally, this referred to code within the codebase of a single application, but with Composer, code can now be shared between applications as well. DRY is another way of saying <em>don’t re-invent the wheel</em>; if someone else has already written code that does what you want to do, rather than writing code that does the same thing, it’s better to re-use the code that that has already been written. For example, the current standard for authentication (aka logging in) to remote systems is the <em>OAuth 2</em> protocol. This is a secure protocol that allows sites or applications to authenticate with other sites, such as Facebook, Google, Twitter, Instagram, and countless others. Writing OAuth 2 integrations is tricky, as the authentication process is somewhat complex. However, other developers have written code that handles OAuth 2 integration, and they have released this code on the internet in the form of a <em>library</em>. A library is basically a set of code that can be re-used by other sites. Using Composer, developers can include this library in a project, and use it to authenticate to the remote API, saving the developer from having to write that code.</p> <p>Composer allows developers to do the following:</p> <ul><li>Download and include a library into a project, with a single command</li> <li>Download and include any libraries that library is dependent upon</li> <li>Check that system requirements are met before installing the library</li> <li>Ensure there are no version conflicts between libraries</li> <li>Update the library and its dependencies with a single command </li> </ul><h2>So how does Composer work?</h2> <p>Composer itself is a software/program. After a user has installed Composer, they can then say ‘Composer: download Library A to my system’. Composer searches remote <em>repositories</em> for libraries. A repository is a server that provides a collection of libraries for download. When Composer finds Library A in a repository, it downloads the library, as well as any libraries that Library A is dependent upon.</p> <div class="well">A note on terminology <p>In this article, the term <em>Library</em> is used. Libraries are also known as <em>Packages</em>, and referred to as such on <a href="https://getcomposer.org/">https://getcomposer.org/</a></p> <p>A <em>project</em> is the codebase, generally for a website or application, that is being managed by Composer. </p></div> <p>By default, the main repository Composer looks at is <a href="https://packagist.org/">https://packagist.org/</a>. This is a site that has been set up specifically for Composer, and contains thousands of public libraries that developers have provided for use. When a user says ‘Composer download Library A’, the Composer program looks for Library A on <a href="https://packagist.org/">https://packagist.org/</a>, the main public Composer repository, and if it finds the Library, it downloads it to your system. If Library A depends upon (aka requires) Library B, then it will also download Library B to your system, and so on. It also checks to make sure that your system has the minimum requirements to handle both Library A and Library B and any other dependencies, and also checks if either of these packages have any conflicts with any other libraries you've installed. If any conflicts are found, Composer shows an error and will not install the libraries until the conflicts have been resolved.</p> <p>While packagist.org is the default repository Composer searches, projects can also define custom repositories that Composer will search for libraries. For example, many developers use Github or Bitbucket, popular services that provide code storage, to store their code in the cloud. A project owner can set up Composer to look for projects in their private Github, Bitbucket, or other repositories, and download libraries from these repositories. This allows for both the public and private code of a project to be managed using Composer.</p> <h2>What happens when I install a library?</h2> <p>Composer manages projects on a technical level using two files: <em>compser.json</em> and <em>composer.lock</em>.  First we’ll look at the <em>composer.json</em> file. This file describes the project. If a developer is using private repositories, the repositories will be declared in this file. Any libraries that the project depends on are written in this file. This file can also be used to set specific folder locations into which libraries should be installed, or set up scripts that are executed as part of the Composer install process. It’s the outline of the entire project.</p> <p>Each library has a <em>name</em>. The name is combined of two parts, first a <em>namespace</em>, which is an arbitrary string that can be anything but is often a company name, or a Github user name etc. The second part is the library name. The two parts are separated by a forward slash, and contain only lower case letters. Drupal modules are all part of the <em>drupal</em> namespace. Libraries are installed using Composer’s <em>require</em> command. Drupal modules can be installed with commands like:</p> <p><code>// Drupal core:<br /> composer require drupal/core<br /> // Drupal module:<br /> composer require drupal/rules<br /> // Drupal theme:<br /> composer require drupal/bootstrap</code></p> <p>When the above commands are run, Composer downloads the library and its dependencies, and adds the library to the <em>composer.json </em>file to indicate that your project uses the library. This means that <em>composer.json</em> is essentially a metadata file describing the codebase of your project, where to get that code, and how to assemble it.</p> <h2>Composer and Git, Multiple Environments and Multiple Developers</h2> <p>Composer and Git work really well with each other. To understand how, let’s first look at traditional site management using Git. Developer A is creating a new Drupal project, purely managed with Git:</p> <ol><li>Developer A downloads Drupal core</li> <li>Developer A creates a new Git repository for the code they have downloaded, and commits the code to the repository</li> <li>Developer A pushes the code to a central repository (often Github or Bitbucket)</li> <li>Developer A <em>checks out</em> (aka pulls) the code to this server.</li> </ol><p>This all sounds good, and it actually works very well. Now let’s imagine that Developer B comes onto the project. Developer B uses Git to download the code from the central repository. At this point, the codebase in Git exists in four locations:</p> <ul><li>Developer A’s computer</li> <li>Developer B’s computer</li> <li>The central repository</li> <li>The production server </li> </ul><p>At the moment, the codebase only consists of Drupal core. The Drupal core code is being managed through Git, which would allow for changes to be tracked in the code, yet it’s very unlikely that either Developer A or Developer B, or indeed any other developers that come on the project, will actually ever edit any of these Drupal core files, as it is a bad practice to edit Drupal core. Drupal core only needs to be tracked by developers who are developing Drupal core, not by projects that are simply using it. So the above setup results in sharing and tracking a bunch of code that is already shared and tracked somewhere else (on Drupal.org).</p> <p>Let’s look at how to start and use Composer to manage a project. Note that this is NOT the best way to use Composer to manage a Drupal site, and is simply an example to show how to use Composer (see part 2 of this series for specifics on how to use Composer to manage a Drupal site).</p> <ol><li>Developer A creates a new project folder and navigates into it.</li> <li>Developer A initializes the project with <code>composer init</code>, which creates a <em>composer.json </em>file in the project folder</li> <li>Developer A adds the Drupal repository at <a href="https://packages.drupal.org/8">https://packages.drupal.org/8</a> to <em>composer.json</em>, so that Drupal core, modules and themes can be installed using Composer</li> <li>Developer A runs <code>composer require drupal/core</code>, which installs Drupal core to the system, as well as any dependencies. It also creates <em>composer.lock</em> (which we'll look at further down the article)</li> <li>Developer A creates a new Git repository, and adds <em>composer.json</em> and <em>composer.lock</em> to the Git repository</li> <li>Developer A pushes <em>composer.json</em> and <em>composer.lock</em> to the central repository</li> <li>Developer A sets up the production server, and <em>checks out</em> the code to this server. At this point, the code consists only of the <em>composer.json</em> and <em>composer.lock </em>files. Additional servers can be set up by checking out the code to any server.</li> <li>Developer A runs <code>composer install</code> on the production server. This pulls all the requirements and dependencies for the project as they are defined in <em>composer.json</em></li> </ol><p>Now when Developer B comes on the project, Developer B uses Git to download the codebase to their local computer. This codebase contains only <em>composer.json</em> and <em>composer.lock.</em> However, when they run <code>composer install</code> they will end up with the exact same codebase as the production server and on Developer A’s machine.</p> <p>Now the codebase exists in the same four locations, however the only code being tracked in the Git repository is the two files used to define the Composer managed project. When an update is made to the project, it is handled by running <code>composer update drupal/core</code>, which will update both <em>composer.json</em> and <em>composer.lock</em>. These files are then updated in the Git repository, as they are the files specific to our project.</p> <p>The difference between the traditional Git method, and the above method using Composer, is that now Drupal core is considered to be an external library, and is not taking up space unnecessarily in our project's Git repository.</p> <h2>Project Versions</h2> <p>Projects can, and pretty much always do, have versions. Drupal 8 uses semantic versioning, meaning that it goes through versions 8.1, 8.2, 8.3… and so on. At the time of writing the current version is 8.6.3. If a new security fix is released, it will be 8.6.4. In time, 8.7.0 will be released.  Composer allows us to work with different versions of libraries. This is a good thing, however it opens up the risk of developers on a project working with different versions of a library, which in turn opens up possibility of bugs. Composer fortunately is built to deal with versions, as we will look at next.</p> <h2>Tracking Project Versions</h2> <p>So how does Composer handle versions, allowing developers to ensure they are always using the same library versions? Welcome the <em>composer.lock</em> file. The <em>composer.lock</em> file essentially acts as a snapshot of the all the versions of all the libraries managed by <em>composer.json</em>. Again, I’ll refer back to the Composer managed site described above. When we first run <code>composer require drupal/core</code> in our project, a few things happen:</p> <ol><li>The current (most recent) version of Drupal is downloaded to the system</li> <li>All libraries that Drupal depends on are also downloaded to the system</li> <li><em>composer.json</em> is updated to show that Drupal is now a dependency of your project</li> <li><em>composer.lock</em> is created/updated to reflect the current versions of all Composer managed libraries</li> </ol><p>So <em>composer.json</em> tracks which libraries are used, and <em>composer.lock</em> is a snapshot tracking which versions of those libraries are currently being used on the project. </p> <h2>Synchronizing Project Versions</h2> <p>The problem with developers using different versions of libraries is that developers may write code that only works on the version of the library that they have, and other developers either don’t yet have, or maybe they are using an outdated version of the library and other developers have updated. Composer projects manage library versions using the commands <code>composer install</code> and <code>composer update</code>. These commands do different things, so next we'll look at the differences between them.</p> <h2>Composer Install and Composer Update</h2> <p>Imagine that Composer didn’t track versions. The following situation would happen (again, this is NOT how it actually works):</p> <ol><li>Drupal 8.5.6 is released.</li> <li>Developer A creates a new project, and sets Drupal core as dependency in <em>composer.json</em>. Developer A has Drupal 8.5.6</li> <li>Drupal 8.6.0 is released</li> <li>Developer B clones the Git project, and installs the codebase using <code>composer install</code>. Composer downloads Drupal core. Developer B has Drupal 8.6.0</li> </ol><p>The two developers are now working on different versions of Drupal. This is dangerous, as any code they write/add may not be compatible with each other's code. Fortunately Composer can track libraries. When a user runs <code>composer install</code>, the versions defined in <em>composer.lock</em> are installed. So when Developer B runs <code>composer install</code>, Drupal 8.5.6 is installed, even though Drupal 8.6.0 has been released, because 8.5.6 is listed as the version being used by the project in <em>composer.json. </em>As such, developers working on Composer managed projects should run <code>composer install</code> each time they pull updates from remote Git repositories containing Composer managed projects.</p> <h2>Updating versions</h2> <p>As has been discussed, the <em>composer.lock</em> file tracks the versions of libraries currently used on the project. This is where the <code>composer update</code> command comes in. Let’s review how to manage version changes for a given library (this is how it actually works):</p> <ol><li>Drupal 8.5.6 is released.</li> <li>Developer A creates a new project, and sets Drupal core as dependency. The <em>composer.lock</em> file records the version of Drupal core used by the project as 8.5.6.</li> <li>Drupal 8.6.0 is released</li> <li>Developer B clones the Git project, and installs the codebase using <code>composer install</code>. The composer.lock file lists the version of Drupal core being used on the project as 8.5.6, so it downloads that version.</li> <li>Developer A sees that a new version of Drupal has been released. Developer A runs <code>composer update drupal/core</code>. Composer installs Drupal 8.6.0 to their system, and updates <em>composer.lock</em> to show the version of Drupal core in use as 8.6.0.</li> <li>Developer A commits this updated <em>composer.lock</em> to Git, and pushes it to the remote repository. </li> <li>Developer B pulls the Git repository, and gets the updated <em>composer.lock</em> file. Developer B then runs <code>composer install</code>, and since the version of Drupal core in registered as being used is now 8.6.0, Composer updates the code to Drupal 8.6.0.</li> </ol><p>Now Developer A and Developer B both have the exact same versions of Drupal on their system. And still the only files managed by Git at this point are <em>composer.json</em> and <em>composer.lock</em>.</p> <h2>Tying it all together</h2> <p>Developers should always run <code>composer.install</code> any time they see that a commit has made changes in the <em>composer.lock</em> file, to ensure that they are on the same codebase as all other developers. Developers should also always run composer.install anytime they switch Git branches, such as between a production and a staging branch. The dependencies of these branches may be very different, and running <code>composer install</code> will update all dependencies to match the current <em>composer.lock</em> snapshot. The <code>composer update</code> command should only be used to update to new versions of libraries, and the <em>composer.lock</em> file should always be committed after running <code>composer update</code>. Finally, any time a developer adds a new dependency to the project, they need to commit both the <em>composer.json</em> file and the <em>composer.lock</em> file to Git.</p> <h2>Summary</h2> <p>Before moving on to the next blog post in this series, you should understand the following:</p> <ul><li>What the <em>composer.json</em> file does</li> <li>What the <em>composer.lock</em> file does</li> <li>When to use <code>composer install</code></li> <li>When to use <code>composer update</code></li> <li>How Git and Composer interact with each other</li> </ul><p>In <a href="/blog/drupal-and-composer-managing-drupal-8-site-composer">Part 2</a>, we'll look specifically at building and managing a Drupal project using composer.</p> Mon, 26 Nov 2018 09:58:58 +1100 Jay Friendly https://www.morpht.com/blog/drupal-and-composer-part-1-understanding-composer