Easily load data into your Scratch Org (or Sandbox): Part 2

Fabien Taillon
Texeï
Published in
4 min readMay 13, 2020

--

This is a two parts blog series: Part 1 is focused on a simple command to export/import data, while Part 2 handles with more advanced use cases.

Limitations

Having an easy to use command to export/import data is nice, but at some point you may be stuck with more complex use cases.

This happened to me with a quite basic one though. The command described in the first part of this blog post series (see here if you haven’t read it) was intended to be as easy to use as possible. But what if you need to handle a relationship between the same sObject ? For instance Account and Parent Account. Well, it just doesn’t work.

We started to use the command on several projects (including moving data for Salesforce CPQ / Billing), and it quickly became obvious that even though the command was easy to use, it had too many limitations:

  • No way to handle relationships on the same sObjects (Account/Parent Account)
  • No way to exclude fields (for instance a source org had audit fields writeable, whereas the target org did not, so we wanted to remove CreatedDate, LastModifiedDate, etc…)
  • No way to update records previously inserted (if you have circular dependencies, you may need to first create a record and update later once some other related records are created)
  • No way to filter the records you want to export

Again, this was something intended at the beginning, as the main point of the command was to be as easy as possible. There are already lots of powerful data loading tools out there that can do whatever you want, and the point wasn’t to replace them.

Still, there was room for improvement.

A more powerful command

So how should we add these features to the existing command ? Exporting a few sObjects looks like this:

sfdx texei:data:export —-objects Account,Contact,Opportunity,MyCustomObject__c --outputdir ./data

Obviously, if we wanted to add filters to each exported sObjects, that would quickly become a giant command to run, and a not very scalable one if we need to add more options (like fields we want to exclude).

We choose in this case to rely on a data plan file, that would describe what we want to export, with every options for every sObjects. The other good thing with using a file is that it makes it easy to export data again with all the same options.

Let’s generate a sample data plan file by running the following command:

sfdx texei:data:plan:generate --objects Account,Contact,MyCustomObject__c --outputdir ./data

This will generate the following data-plan.json file for you (provided that the data folder already exists):

Then you can fill in all options:

  • excludedFields : a list of fields to exclude. Either at sObject level, or at root level where it will exclude fields for all sObjects
  • name : the developer name of the sObject to export. This is the only mandatory field
  • label : a label that will be used for the file, useful if you’re exporting the same sObject several time
  • filters : a filter that will get added to a where clause

Each sObject will be exported and imported in the order coming from the json file, meaning you’ll need the parent sObjects to be listed first, otherwise hierarchy won’t be exported/imported correctly.

Managing relationship between the same sObject

With a single level of hierarchy

Let’s take a basic Account/Parent Account relationship, you could simply run the following:

sfdx texei:data:plan:generate --objects Account,Account --outputdir ./data

Here, Account is exported twice, because we first want to export the parents, and then the children. This way parents will be inserted first, the command will bring the Ids back from the insert, and the relationship will be replaced on the fly while inserting child accounts. Your data plan would just need to filter the ParentId accordingly:

Running the sfdx texei:data:export command with this data plan would just:

  • Export all accounts without a parent (ParentId = null )
  • Export all accounts with a parent ( ParentId != null )

Then running the sfdx texei:data:import command would:

  • Import the parents first (because it’s the first exported sObject)
  • Import the child, making sure that parents are already imported

Doing it the other way round would just insert the child sObjects with the ParentId lookup empty, because…

> > > Click here to read the full article < < <

Click here to access all Texeï blog posts

--

--

Salesforce MVP | CTO @TexeiFrance | @ParisDUG Co-organizer | @FTDreamin Team