Peliqan has ETL built-in for a wide range of connectors. This means that when you add a connection to a source, Peliqan will automatically create pipelines and start syncing the data into the Peliqan built-in data warehouse (DWH) or your own DWH.

However, for some cases it might be useful to implement a custom pipeline in Peliqan, using low-code Python scripts.

Fetch data from a source

A custom pipeline will typically fetch data from some source incrementally:

Incremental processing from source

Write data to the DWH

Next, the data from the source will be written to the DWH. Peliqan has a powerful built-in function dw.write() that will write a dataset to a table in the DWH, and prior to that it will create the table if needed and add columns if needed. More info:

Writing data to tables

Handling deleted rows

A regular ETL pipeline cannot detect if records were deleted in the source. This means that when a record is deleted in a source, it will remain in the DWH. Here are various patterns to handle deleted rows accordingly:

Handling deleted rows