Upgrading Drupal 8 to Drupal 9

Drupal 8 (aka D8) reached end of life last November. The KWLUG website was running Drupal 8, so it was time to upgrade.

I had heard that the Drupal 8 to Drupal 9 (aka D9) transition would not be as much work as our Drupal 6 to Drupal 8 had been. This was true, but the process was not as simple as composer update either. As always, Drupal is opinionated about "The Drupal Way To Do Things (tm)" and any deviations from the Drupal Way cause headaches. The official documentation was okay, but there were still gotchas that tripped me up. Furthermore, it was not always clear how I could contribute to the official documentation so future people would not get tripped up, so I am documenting things here.

Upgrade Status

The first step is to back up the production site and practice the upgrade process on development clones. Trying to do a major version upgrade on a production site is madness.

The Drupal documentation also recommended I install a package called Upgrade Status. I did this on a clone:

composer require drupal/upgrade_status
drush pm-enable upgrade_status

This created an administration menu which showed me the modules I could safely delete from the D8 site. It also analysed the D8 site and made other suggestions about my config files and custom modules.

Composer

I am still no fan of Composer, but it is the Drupal Way. Fortunately, I started with Composer near the beginning of our Drupal 8 experience. Unfortunately, the scaffolding Drupal uses for Composer changed during that time. The official upgrade documentation links to these instructions, which suggest making a lot of complicated changes to composer.json to make it compatible with Drupal 9. This approach did not work for me. I found it easier to clone the drupal/recommended-project templates and then carefully replace my project's composer.json with the cloned one. The composer.json was for Drupal 9.

"Carefully" is an important word here.

First I upgraded to the latest version of Drupal 8 (8.9?) using my old composer.json
Then I carefully looked at the dependencies specified in my composer.json file. I classified them as ones I wanted to keep, and those that were no longer relevant. It was very important to minimize the number of listed dependencies, so if I knew one dependency was a prerequisite of another, I got rid of the prerequisite. In addition, I decided to get rid of all dependencies in the require-dev section.
There were some modules (such as drupal/migrate_csv) which were left over and could safely be removed. First I called drush pm-uninstall on them, then I prepared to omit them from composer.json, and then I moved the module files out of the way.
For each dependency, I looked up the latest stable version for Drupal 9.
I then replaced the existing composer.json with the one from drupal/recommended-project. I added all the dependencies in the replaced composer.json.
I patched my custom code to be compatible with Drupal 9, as documented below.
I crossed my fingers and ran composer update to do the upgrade from Drupal 8 to Drupal 9.
Finally I used drush pm-uninstall to get rid of the modules I did not want any more, and then used composer remove to get rid of them entirely.

I had not changed any of my Drupal 8 composer.json other than to add dependencies. Apparently there are many customizations you can make to it, and obviously if you made such customizations you would have to carry over those changes too.

Here is a concrete example. In my original D8 composer.json I had:

    "require": {
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "~1.0",
        "drupal/core": "^8.7",
        "drush/drush": "^8.0",
        "drupal/console": "~1.0",
        "drupal/migrate_plus": "^3.0",
        "drupal/admin_toolbar": "^1.18",
        "drupal/redirect": "^1.0@alpha",
        "drupal/migrate_upgrade": "^3.0",
        "drupal/riddler": "1.x-dev",
        "drupal/adaptivetheme": "^1.0@RC",
        "drupal/migrate_tools": "^3.0",
        "drupal/migrate_source_csv": "^2.0",
        "drupal/at_tools": "^2.0",
        "drupal/git_deploy": "2.x-dev",
        "drupal/views_fieldsets": "^3.3",
        "drupal/token": "^1.6"
    },
    "require-dev": {
        "behat/mink": "~1.7",
        "behat/mink-goutte-driver": "~1.2",
        "jcalderonzumba/gastonjs": "~1.0.2",
        "jcalderonzumba/mink-phantomjs-driver": "~0.3.1",
        "mikey179/vfsstream": "~1.2",
        "phpunit/phpunit": "~4.8",
        "symfony/css-selector": "~2.8"
    },

Everything inside the require-dev section went away (mostly because phpunit was causing a lot of problems). My new composer.json looked something like this:

    "require": {
        "composer/installers": "^1.9",
        "drupal/core-composer-scaffold": "^9.3",
        "drupal/core-project-message": "^9.3",
        "drupal/core-recommended": "^9.3",

        "drush/drush": "^11.0",
        "drupal/riddler": "^1.2",
        "drupal/adaptivetheme": ">=1.4.x-dev",
        "drupal/views_fieldsets": "^3.4",
        "drupal/token": "^1.10",
        "drupal/admin_toolbar": "^3.1",
        "drupal/redirect": "^1.7",

        "drupal/at_tools": "^3.3",
        "drupal/git_deploy": "^2.3"
    },
    "require-dev": {
    },

The first group (until drupal/core-recommended) came with the composer.json. The second group were the modules I wanted to maintain in the new site, including their currently supported versions. drupal/at_tools and drupal/git_deploy were things I no longer wanted in the site, but I kept them around until after the upgrade. If I got rid of them from composer.json without removing them using drush first, then drush updb database updates would fail.

AdaptiveTheme

I use the AdaptiveTheme theme on kwlug.org . This theme is simple and worked "well enough" for Drupal 8. In upgrading to Drupal 9 there was some drama.

Firstly, the AdaptiveTheme project was seemingly abandoned, and so was forked into at_theme. The original AdaptiveTheme wanted a package called at_tools but the fork needed a fork of at_tools called at_tool, which was incompatible with the original at_tools. What a mess!

Then I learned that the author of the forked theme gained control of the original version of AdaptiveTheme. So maybe I did not need to switch at all, and in fact the upgrade from whatever ancient version I was running to an updated version should have been fine... except for composer.json.

You see, the line in my original composer.json read:

        "drupal/adaptivetheme": "^1.0@RC",

which was okay except that the modern version of AdaptiveTheme was 5.1 and for some reason Composer would not upgrade my installed AdaptiveTheme with a new version. This broke other things, which ended up breaking the Drupal 9 upgrade.

My fix was to explicitly require a version of AdaptiveTheme greater than or equal to 1.4:

        "drupal/adaptivetheme": ">=1.4.x-dev",

This I guess worked? I believe this installed version 4.1 of AdaptiveTheme, which I then upgraded to version 5.1 later:

        "drupal/adaptivetheme": "^5.1",

(I wish I had exact error messages to show here, but it looks as if I did not record them. Sorry. You won't even find this web entry because of bad SEO, though.)

Also, pinning requirements using == is a terrible idea. I do not even love using ^ to pin major versions, but apparently that is The Drupal Way.

Custom Modules

The Upgrade Status module detected some small problems in my custom modules (mostly dpm and dsm debugging calls I had forgotten to comment out) but the big change was that I was supposed to add the following line to each of my *.info.yml files:

core_version_reqirement: "^8.7 || 9"

I was not going to redistribute these modules, so I cheated and allowed everything bigger than Drupal 8:

core_version_requirement: ">=8"

No doubt this will burn me in the future.

PHP Version and Drush

According to the official documentation, Drupal 9 requires PHP 7.3. However, there is a caveat: "Some individual modules may have specific requirements for PHP extensions and configurations beyond those listed below, so, please check the module's documentation as well." Sure enough, I got burned.

I do not know whether drush is still The Drupal Way, but I depend upon it pretty heavily and consider it a fundamental part of Drupal. Drupal 9 requires Drush 11, and Drush 11... depends on PHP 7.4 . So if you want to use Drush, you need to upgrade your PHP, or you get messages like:

  Problem 1
    - drush/drush[11.0.0, ..., 11.0.5] require php >=7.4 -> your php version (7.3.33) does not satisfy that requirement.
    - Root composer.json requires drush/drush ^11.0 -> satisfiable by drush/drush[11.0.0, ..., 11.0.5].

Drupal actually recommends PHP 8.0 or higher, but my webhost only provided up to PHP 7.4 . Fortunately I was able to switch my PHP version to that, and enable all the other PHP modules that Drupal requires.

The other headache (as always) is getting Drush into your $PATH. To do this I symlink Drush from the vendor folder:

ln -s vendor/drush/drush/drush ~/bin

but this is not the right way to do it, because it only works for one installation.

Drush Backups

drush archive-dump has been removed from Drush, because it does not back up the vendor or script directories. This is yet another casualty of Composer.

My workaround was to use drush sql:dump to grab the database, and then use plain old tar to grab the entire folder containing all of Drupal. Grabbing composer.json is good too.

In principle, if you have the following you should be able to restore the entire Drupal site:

Your custom modules and themes
The sites/default folder, which contains static files and your settings.php stuff.
Your composer.json
Your database dump

Although this ought to work, I have not verified it.

Content Type Reinstallation

I made a big mistake when migrating my Drupal site from Drupal 6 to Drupal 8. I had a bunch of custom content types I wanted in the new site, so I made a custom module (kwlug_content_types) to store the YAML files for those types. In my custom module I had a config/install folder which installed all those content types.

In addition, I had some custom code that would interact with some of the custom types I created. I put this in the kwlug_content_types module as well.

Here is the problem. If I ever uninstall the kwlug_content_types module then I am hosed, because I cannot reinstall it -- if I try, then I get lots of errors about content types existing already. But if I do not reinstall the module then I lose my custom code.

My solution to this was a bad solution: I moved the YAML files from config/install to config/optional. Then if the content types already existed in the Drupal install they would not be overwritten. This is terrible because if some obsolete version of a content type existed already, it would not be fixed with the install.

I do not know The Drupal Way to handle this. I am thinking that maybe splitting the module into YAML files separate from the custom code may have been wiser.

Config Files

Before doing the upgrade I needed to make the web/sites/default folder user-writable, and then I needed to fix it.

In the settings.php I had to change $config_directories['sync'] to $settings['config_sync_directory'] . I got this information from the Lullabot upgrade tutorial.

Group Readership

In my development sites I had my files owned by my user account, but the web server being run as nginx. Thus I had to add nginx to my group account, but this caused lots of headaches. I needed to make lots of things group writeable, and then composer install overwrote those permissions and I needed to fix things again. There is probably some umask magic that would have fixed things, but I ran out of motivation before I figured it out. Fortunately this was not an issue on my production site.