Salesforce Data Migration Orchestration

As discussed in the previous tutorial, orchestration is the management and execution of the individual Jobs that will make up your entire data migration. This orchestration will manage the individual steps in your data migration from data acquisition through to data loading and migration reporting.

There may be several layers of orchestration. Once you’ve got to the data loading stage, for example, you will then orchestrate the individual data loading Jobs.

Managing your data migration in this way will ensure that it is robust and, should there be a failure, that you will be able to restart the data migration from that point of failure (see Checkpointing).

Job: SalesforceMigrateOrchestrate

This is the main orchestration Job and will orchestration the data migration from data acquisition through to data loading and migration reporting.

The following, lists the key tasks that will be orchestrated: –

  • Acquisition
  • Identification
  • Profile
  • Quality Control
  • Preparation
  • Finanlisation
  • Load
  • Migration Reporting

Remember that, in all but the most trivial of data migrations, these tasks will not always be linear. You may, for example, have multiple points where you need to perform Quality Control and, with dependencies on loading, you may have a number of Finalisation and Load phases. This subject will be discussed in more detail, later.

Initialisation

As with most Jobs that you create within the Framework, Job: SalesforceMigrateOrchestrate is based on the Framework template Job: TemplateJob. As can be seen from the following screenshot, amended components have their RunJob color changed to orange, as well as being correctly labelled, if required.

Salesforce Data Migration, Orchestration, Initialisation
Salesforce Data Migration, Orchestration, Initialisation

Data Acquisition Checkpoint

The following screenshot shows the first of the Checkpoints, for Data Acquisition.

Salesforce Data Migration, Orchestration, Data Aquisition
Salesforce Data Migration, Orchestration, Data Aquisition

Checkpointing allows us to have controlled recovery from any point within our processing. If we have successfully completed Data Aquisition and our processing subsequently fails, we do not need to re-run Data Aquisition once we have resolved the underlying issue and restarted our Job: SalesforceMigrate.

We may, of course, have additional Checkpointing within Job: SalesforceMigrateAcquire. If a Checkpoint fails, we may choose to re-run the Checkpointed logic in its entirety, or have some additional logic that performs an alternative recovery procedure.

We will see this same pattern of Checkpointing throughout the entire migration Job, from Data Acquisition through to Data Loading and Migration Reporting.

Next Buton