Texas Camp: Migration - How to Tame the Data

Submitted by Steven on Sat, 04/30/2016 - 15:17
Texas Camp

 

Migration has become the standard method to upgrade Drupal sites between major versions and external sources. In this session you will learn how to set up a migration and how to overcome common gotchas. Finally you will learn the current state of migration in Drupal 8 and the differences from Drupal 7. You will leave this session with an improved understanding of how to approach migrations and how to plan for the road ahead.

Methods

  • Drupal API
  • Feeds
  • Migration module (d7)
  • Migrate API (d8)

Method: Drupal API

  • Write custom code to parse files or connect to database.
  • Use drupal api like, node_save(), user_save(), taxonomy_term_save(), taxonomy_vocabulary_save(), entity_create with entity_metadata_wrapper.
  • Advantage: You are in total control. Not waiting on contrib modules.
  • Disadvantage: Time consuming
    • Write your own database query or file parser.
    • Write your own loops.
    • Detailed preparation of each object and its settings and fields.
    • Lots of opportunity to make typos and mistakes.
    • No easy way to roll back.
    • Manage large imports by incorporating Batch API.

Method: Drupal 7 API - create node

D7 Node

Method: Drupal 8 API - create node

Method: Drupal 7 API - create terms

Method: Drupal 8 API - create terms

Method: Drupal 7 API - create user

Method: Drupal 8 API - create user

Method: Feeds

  • UI for importing content without writing code.
  • Several parsers to extend feeds.
    • Default: RSS, CSV, OPML, Sitemap
    • XPath XML
    • QueryPath XML
    • JSONPath
    • JMESPath (JSON)
    • SQL
  • Feeds Tamper: UI plugin for altering data.
  • Does provide hooks for data alteration.

Method: Feeds hook_feeds_after_parse()

Alter data before it is processed.

Example of altering a country codes.

Altar just before pre-save.

Method Feeds: Demo

Demo Current D8 Feeds Interface

Method: Migration Module (d7)

  • Define migration in Object oriented code.
  • Extend migration classes in custom module (developers tool)
  • Source support for PDO, XML, CSV, JSON, MSSQL, ORACLE
  • Destination support for node, user, taxonomy term, comment, file.
  • Map sources to destination.
  • Manage dependencies.
  • UI interface to track mapping of fields and perform migration.
  • Roll back support for easy development and testing.
  • Memory management for large running migration.
  • Alter data with prepare before it is saved and complete after saved.
  • Extend with extra modules: migrate_d2d, wordpress_migrate

Method: Migration Module (d7)

  • Register your custom class
  • Define your class that extends a migration class.
  • Define key mapping
  • Define columns for mapping.
  • Define source
  • Map fields
  • Unmap source fields and destination fields.
  • Perform operations if needed on prepare and complete.

Method: Migrate extending Migration

Method: Migrate API (d8)

Migration API added to d8 core with several contrib modules

  • Migrate Core: Provides Core API
  • Migrate Drupal: Provides classes to migrate configuration and content d6->d8 & d7->d8.
  • Drupal Upgrade: Contrib being added to core in 8.1.x
  • Migrate Tools: provides optional Drush commands
  • Migrate Plus: Extends API with grouping, source manipulation and examples (Beer).
  • Migrate Manifest: Provides drush command migration from manifest file.

Method: D6/D7 to D8 Issues

  • D6 has Story type vs Article type
  • Views migration: Views 6.x-2.x needs to upgrade to 6.x-3.x
  • Date Formats: Only the default short, medium and long formats are migrated.
  • Text/Input Formats: if input format is not recognized it will be imported as filter_null
  • Timezones: D6 uses timezone offset. D7 & D8 use a timezone name. Could cause problems depending on server setup.
  • Node, User, Entity Reference fields not supported.
  • Multilingual Content
  • D6 to D8 supports:Core, CCK, Link, Email, Phone, Imagecache modules
  • D7 to D8 supports:Only content, users, taxonomy, blocks, menus, filters

Method: Migrate Extended

  • To customize a d6 migration you will need to create your own module.
  • Include the proper Namespace and classes.
  • Create a migration template
    • Reference: core/modules/node/migration_templates
  • Create your own template with your own ID and source id.

Method: Migrate Extended

  • Create your PHP file to extend D6node.
  • Include the proper namespaces and classes
  • Update the MigrateSource id to match your template migrate source id.
  • Use function fields() to add fields.
  • Use function query() to access the database.
  • Use function prepareRow(Row $row) to prepare data.

Method: D6/D7 to D8

UI

  • Enable appropriate module and navigate to /upgrade in D8
  • Rerun
  • Rollback

Drush

  • drush migrate-upgrade --legacy-db-url=mysql://user:@127.0.0.1/drupal6
  • With migrate_tools you can use additional drush commands
    • Drush help to view options

Method Migrate - D6 to D7 Demo

D8 node