Colorado Camp: Migration - How to Tame the Data

Submitted by Steven on Mon, 06/22/2015 - 14:44
Drupalcamp Colorado 2015


View session Slides for: Migration - How to tame the Data

Migration has become the standard method to upgrade Drupal sites between major versions and external sources. In this session you will learn how to set up a migration and how to overcome common gotchas. Finally you will learn the current state of migration in Drupal 8 and the differences from Drupal 7. You will leave this session with an improved understanding of how to approach migrations and how to plan for the road ahead.

Migrating data into Drupal 7 and the current state of Drupal 8.


  • Drupal API
  • Feeds
  • Migration module (d7)
  • Migrate API (d8)

Methods: Drupal API

  • Write custom code to parse files or connect to database.
  • Use drupal api like, node_save(), user_save(), taxonomy_term_save(), taxonomy_vocabulary_save(), entity_create with entity_metadata_wrapper.
  • Advantage: You are in total control. Not waiting on contrib modules.
  • Disadvantage: Time consuming
    • Write your own database query or file parser.
    • Write your own loops.
    • Detailed preparation of each object and its settings and fields.
    • Lots of opportunity to make typos and mistakes.
    • No easy way to roll back.
    • Manage large imports by incorporating Batch API.

Method: Drupal API - create node

Drupal API node create

Method: Drupal API - create terms

Drupal API Term Create

Method: Feeds

  • UI for importing content without writing code.
  • Several parsers to extend feeds.
    • Default: RSS, CSV, OPML, Sitemap
    • XPath XML
    • QueryPath XML
    • JSONPath
    • JMESPath (JSON)
    • SQL
  • Feeds Tamper: UI plugin for altering data.
  • Does provide hooks for data alteration.

Method: Feeds hook_feeds_after_parse()

Alter data before it is processed.
Example of altering a country codes.

Hook feeds after parse

Method: Feeds hook_feeds_presave()

Altar just before pre-save.

Hook feeds presave

Method Feeds: Demo

Demo Feeds CSV, SQL import

Method: Migration Module (d7)

  • Define migration in Object oriented code.
  • Extend migration classes in custom module (developers tool)
  • Source support for PDO, XML, CSV, JSON, MSSQL, ORACLE
  • Destination support for node, user, taxonomy term, comment, file.
  • Map sources to destination.
  • Manage dependencies.
  • UI interface to track mapping of fields and perform migration.
  • Roll back support for easy development and testing.
  • Memory management for large running migration.
  • Alter data with prepare before it is saved and complete after saved.
  • Extend with extra modules: migrate_d2d

Method: Migration Module (d7)

  • Register your custom class
  • Define your class that extends a migration class
  • Define key mapping
  • Define columns for mapping
  • Define source
  • Map fields
  • Unmap source fields and destination fields
  • Perform operations if needed on prepare and complete

Method: Migrate extending Migration

Extend migration class

Method: Migrate Demo

Demo migrate module

Method: Migrate API (d8)

Migration API added to d8 core.

  • No rollback
  • No UI, drush only
  • Limited to PDO for source.
  • Uses yaml config files.
  • Build in support for : migrating d6->d8 & d7->d8.
  • Follow IMP group for progress or contrib:

Method: Migrate - Drupal Update (d8)

To perform a migrate from d6->d8 create a config file.

  • No UI, perform with drush: migrate_upgrade: project to provide UI.
  • Get list of available migrations: drush config-list|grep migrate
  • Create “manifest.yml” and place in root.
  • drush help migrate-manifest
  • drush migrate-manifest --legacy-db-url=mysql://root:@ manifest.yml

Method Migrate - Manifest Demo

DEMO d6->d8 Migration.