01 - How Restores Work

01 - How Restores Work

At a high level, when Universal Migrator restores data, it connects to APIs at the destination system and rebuilds records based on information in the backup.

Data Layering

When Universal Migrator restores a record, it does so by "layering" information into the destination system.  For example, when processing contacts, Universal Migrator might do the following:
  1. First, create 'empty' contacts that just consist of names
  2. Then, layer on email addresses, phone numbers, and postal addresses.
  3. Then, layer on employer/employee hierarchies.
  4. Then, layer on custom field values.
While some systems may support bulk/aggregate APIs, Universal Migrator intentionally does not use those APIs because they tend to be fragile and actually increase the total time of the migration.

All bulk APIs are different, but in general, they tend to respond in similar ways which are not good for data migrations.

For example, lets say you want to restore a batch of 100 contacts and a single contact has a single custom field value that is invalid.  Bulk APIs tend to respond in one of the following ways:
  1. "One of the contacts had an invalid value.  So we didn't process any of them".
  2. "We processed 99 contacts successfully but one completely failed.  Go figure out which one errored and why".
  3. "We processed everything you wanted but one field didn't go through right.  Here's a warning message that requires a human to read and figure out what went wrong where."
By layering the data, if there is an issue, Universal Migrator gains deep insight into the cause and source and enables you to tweak and retry the restore for that record.

Layering is Optimized 

Each destination system will layer data in the order that is ideal for that system.  Our developers take a number of factors into account when deciding what data can be layered concurrently and have generally organized the layers to do the following:
  1. Ensure all dependencies should be restored first (ie.  Matters will never restore before Contacts).
  2. Minimize the total duration of the migration
  3. Transfer documents as quickly as possible
  4. Balance Migration Workstation system resources so there are not too many record types restoring at once.
  5. Leverage all available API limits
  6. Avoid server-side race conditions by not adding to/editing the same object from multiple threads at once.
Info
For most migrations, you should not try to influence the order of layers.  We know best and we've built that intelligence into Universal Migrator.

For example, if you tell Universal Migrator to restore all records types into "System X", the records may load in the following order:
  1. Layer 1: Record types that have no dependencies
    1. Practice Areas
    2. Custom Field Definitions
    3. Document Categories
    4. Contacts
  2. Layer 2: Record types that only depend on record types from Layer 1
    1. Matters
  3. Layer 3: Record types that only depend on record types from Layers 1-2
    1. Documents
  4. Layer 4: Record types that only depend on record types from Layers 1-3
    1. Contact Email Addresses
    2. Contact Phone Numbers
  5. etc.
When you look at the order above, you will notice that our developers could have choose for Contact Email Addresses and Contact Phone Numbers to load in Layer 2 but chose to put it in Layer 4 instead.  This is because we believe it is important for documents to transfer as quickly as possible: if a migration takes a long time, it is much more annoying to need documents from a legacy system than to need a contact's email address.  Therefore, get documents transferring as quickly as possible.

If you want to override that behavior and influence the order, just restore twice with different options.  For example:
  1. Options for Restore 1:
    1. Restore Practice Areas, Document Categories, Contacts, Matters, Contact Email Addresses, Contact Phone Numbers, and Contact Postal Addresses
  2. Options for Restore 2:
    1. Restore Documents

Automated Tweaks

When records are restoring, Universal Migrator may automatically make minor tweaks depending on requirements in the destination system.  For example, if a destination does not allow matters with an empty description, Universal Migrator may provide a default value automatically.  Universal Migrator will never change or ignore wrong data - its automated tweaks are limited to ones that are minor and do not affect the correctness of the record that is restored.

Successfully Restoring a Record

In order for a record to successfully restore into a destination system:
  1. All required dependencies:
    1. Must exist inside the migration database
    2. Must have a valid, non-blank Result_Id value
  2. All optional dependency values:
    1. Must be blank or exist inside the migration database
    2. Must have a valid, non-blank Result_Id value
  3. Any Destination-Specific requirements must be met.
Only dependencies that are used by the destination system are validated.
For example, if a matter is linked to an invalid matter stage but the destination system does not support matter stages, then

As an example, assume we have a matter with the following values:
ColumnValueImplied Rules
Final_ClientContact_IdADEOUIIf the destination system supports contacts, the contact with Id 'ADEOUI' must have a valid Result_Id
Final_PracticeArea_Id
If the destination system supports practice areas, it must allow matters with no practice areas.
Final_MatterStage_IdUEOAIf the destination system supports matter stages, the matter stage with Id 'UEOA' must have a valid Result_Id.
Final_ReferenceCodeABC.001If the destination system supports matter numbers, it should try to create a matter with ABC.001 as the matter number.
Final_Subject
If the destination system supports matter names, it should try to use a blank name if possible and if blank names are not supported, it should use a default value instead.
Final_Description2019 Divorce with ChildrenIf the destination system supports matter descriptions, it should try to use '2019 Divorce with Children'.

Common Reasons Records Fail to Restore

Missing Records

Some legacy systems do not maintain referential integrity between records. This means that it is possible for Matter #15 to be linked to Contact #10, but Contact #10 might no longer exist. If you backed up one of these systems, you must do one of the following:
  1. Have the client fix the issue in their legacy system and then re-backup
    OR
  2. Create a new Contact #10 in the migration database with the matching primary key value of 10.
    OR
  3. Update Matter #15 to reference a contact that does exist using a query similar to the following:

    1. --Update just Matter #15
    2. UPDATE __M_Matters SET Final_ClientContact_Id ='11' WHERE Id = '15' 


    5. --or Update all matters linked to Client #10
    6. UPDATE __M_Matters SET Final_ClientContact_Id ='11' WHERE Final_ClientContact_Id = '10'

Not-Yet-Restored Records

Universal Migrator inherently knows that certain record types must be restored in certain orders.  This means, for example, that staff members, practice areas, and contacts will be restored before matters because matters can have dependencies on these types of data.
 
In the event that there is an error restoring a "parent" record, any items that depend on that record will not be able to be restored until the issue is resolved.  For example, if Matter #15 is linked to Contact #10 but an error occurred restoring Contact #10, then Matter #15 will error and indicate that it could not complete because it was missing the client.


    • Related Articles

    • 01 - How Backups Work

      When you look at all the different CRM, practice management, and document management systems, they all store and access data differently. Some are cloud based, others are on-premise. Some expose JSON APIs, while others use any number of database ...

    • 03.04 - Scoping Topics: How long does it take? How long will I be unable to work?

      This assumes the firm is doing a migration on Thursday per our recommendation. How long does it take? When firms ask how long a migration takes, what they usually are trying to ascertain is "how long will I be unable to work?". It is important to ...

    • 03 - How We Expect Consultants to Work

      The team at Universal Migrator supports over 200 data and document migrations every single month. These migrations are performed by consultants worldwide who are servicing law firms of different sizes with varying needs and requirements. The training ...

    • Training: Overview

      Welcome to the Migration Academy! This series will help you learn everything you need to know in order to deliver high-value, rock-solid data and document migrations. Completion of the academy generally takes 4-8 hours and is required in order to ...

    • 03 - Scoping your First Migration

      When a patient needs an operation, a good surgeon gets as much information as possible from the patient before recommending a course of action or providing a quote. By taking this approach, the surgeon can provide a better, more informed experience ...