As discussed in the previous tutorial, orchestration is the management and execution of the individual Jobs that will make up your entire data migration. This orchestration will manage the individual steps in your data migration from data acquisition through to data loading and migration reporting.
There may be several layers of orchestration. Once you’ve got to the data loading stage, for example, you will then orchestrate the individual data loading Jobs.
Managing your data migration in this way will ensure that it is robust and, should there be a failure, that you will be able to restart the data migration from that point of failure (see Checkpointing).
This is the main orchestration Job and will orchestration the data migration from data acquisition through to data loading and migration reporting.
The following, lists the key tasks that will be orchestrated: –
- Quality Control
- Migration Reporting
Remember that, in all but the most trivial of data migrations, these tasks will not always be linear. You may, for example, have multiple points where you need to perform Quality Control and, with dependencies on loading, you may have a number of Finalisation and Load phases. This subject will be discussed in more detail, later.
As with most Jobs that you create within the Framework, Job: SalesforceMigrateOrchestrate is based on the Framework template Job: TemplateJob. As can be seen from the following screenshot, amended components have their RunJob color changed to orange, as well as being correctly labelled, if required.
Data Acquisition Checkpoint
Checkpointing allows us to have controlled recovery from any point within our processing. If we have successfully completed Data Aquisition and our processing subsequently fails, we do not need to re-run Data Aquisition once we have resolved the underlying issue and restarted our Job: SalesforceMigrate.
We may, of course, have additional Checkpointing within Job: SalesforceMigrateAcquire. If a Checkpoint fails, we may choose to re-run the Checkpointed logic in its entirety, or have some additional logic that performs an alternative recovery procedure.