Data sync are patterns to sync data between a source and a target. In many cases the source and target are both SaaS applications, for example keeping Customer records in sync between a CRM and an accounting system.

Data syncs are implemented in Peliqan by writing low-code Python scripts (”Data apps”) that are scheduled to run for example every 5 minutes (near real-time), every hour or every day. These scripts are also referred to as “Custom data pipelines” and they use the Peliqan SaaS API wrapper functions to do so called writebacks to a target.

Patterns for data sync

Peliqan provides a wide range of features to implement robust, stable and efficient data syncs, by offering a set of patterns to implement these data syncs.

The basic pattern is to read data from a Source, and write the data in destination or Target.

From the Source, the data needs to be read incrementally.

In the Target, the data needs to be written using an upsert.

Data warehouse as central hub for data syncs

The patterns in Peliqan use the data warehouse (DWH) as a central data hub, which means the source is usually the data warehouse (which holds the data from the source SaaS application).

Below is an example that syncs data from a CRM source to an Accounting target. First you add a connection to the source in Peliqan. This will automatically create an ETL pipeline and sync the data from the source into the DWH. Your data sync scripts read from the DWH tables and write to the target.

Note that Peliqan will also sync the data from the target to the DWH (in grey in the picture). This allows for verification of the data in the target, e.g. check if all additions or updates were executed correctly, if no data is missing etc.

More info

Incremental processing from source

Avoiding duplicates (upsert in target)

Keeping track with “link tables”

Parent child relations