08 - Special Topics: Restore Performance and Threading

When Universal Migrator transfers data, it takes a number of precautions to ensure that bad things do not happen in the destination system.

Threading

Universal Migrator uses multiple threads to transfer data concurrently.  Each thread is assigned a group of records to process and will process each item in that group one at a time.   When a thread finishes its group, the thread will request another group to start working on.

Grouping is Per-Parent

Many destination systems have race conditions that can trigger when lots of objects are created with the same parent at the same time.

To avoid issues like this, Universal Migrator will group objects based on their most logical "parent".  For example, matters are often grouped by their Final_ClientContact_Id.

Lop-Sided Data

If you are working with a small amount of data or lop-sided data, you may inadvertently cause Universal Migrator to operate in single-threaded mode.  For example, let's say you want to migrate all of the documents for a single matter.  Because threads are assigned per-parent, only one thread will be working because there is only one group worth of records to process.

Some consultants inadvertently create lop-sided data when doing test migrations.

Big Groups Get Assigned Threads First

Unless you've used Action_Group and Action_Priority to customize the order of data, groups with lots of records are transferred first.  For example, if matters are transferring and there is a client with 1000 matters, it will receive a processing thread sooner than a client with a single matter.  Universal Migrator does this so that it can use maximum concurrency throughout as much of the migration as possible and avoid operating in single-threaded mode.

Destination API Limits

On systems that implement API limits or throttling, Universal Migrator will watch the HTTP response headers and automatically adjust its rate so as to not overload the destination system.

• Related Articles

• 07 - Special Topics: Migrating from Slow Providers

  Universal Migrator's best practices are generally optimized around doing migrations as quickly as possible. This is the ideal scenario, however, sometimes providers implement restrictions that force data to move much slower than technologically ...

• 01 - Tweaking a Restore

  Sometimes after restoring a backup, a customer may want you to tweak the data that is in their new system. Tweaking data post-restore is easy! Let's say that the customer wants you to "mark as complete" all tasks that are related to archived (closed) ...

• 03 - Pre-Restore Processes

  Between the time that you took your initial backup and wrote your playbook, some time will have elapsed. During this time, the client has likely added or changed data in their source system. Prior to completing the restore, you should do the ...

• 04 - Pre-Restore Review

  All Consultants Prior to restoring data into a destination system, you should thoroughly review and test your playbook. Your playbook should be complete and cover all the data that you expect to migrate as part of your project. Thoroughly review and ...

• 02 - Restore-Specific Database Columns

  The following columns are commonly used when Universal Migrator is restoring data into a destination system. Restore Ordering and Priority Action_Group The Action_Group allows you to mass-segment data. Only the data with the highest Action_Group in ...