08 - Special Topics: Restore Performance and Threading

08 - Special Topics: Restore Performance and Threading

When Universal Migrator transfers data, it takes a number of precautions to ensure that bad things do not happen in the destination system.

Threading

Universal Migrator uses multiple threads to transfer data concurrently.  Each thread is assigned a group of records to process and will process each item in that group one at a time.   When a thread finishes its group, the thread will request another group to start working on.

Grouping is Per-Parent

Many destination systems have race conditions that can trigger when lots of objects are created with the same parent at the same time.

To avoid issues like this, Universal Migrator will group objects based on their most logical "parent".  For example, matters are often grouped by their Final_ClientContact_Id.

Lop-Sided Data

If you are working with a small amount of data or lop-sided data, you may inadvertently cause Universal Migrator to operate in single-threaded mode.  For example, let's say you want to migrate all of the documents for a single matter.  Because threads are assigned per-parent, only one thread will be working because there is only one group worth of records to process.

Warning
Some consultants inadvertently create lop-sided data when doing test migrations.

Big Groups Get Assigned Threads First

Unless you've used Action_Group and Action_Priority to customize the order of data, groups with lots of records are transferred first.  For example, if matters are transferring and there is a client with 1000 matters, it will receive a processing thread sooner than a client with a single matter.  Universal Migrator does this so that it can use maximum concurrency throughout as much of the migration as possible and avoid operating in single-threaded mode.

Destination API Limits

On systems that implement API limits or throttling, Universal Migrator will watch the HTTP response headers and automatically adjust its rate so as to not overload the destination system.