Peliqan automatically generates lineage for tables and SQL queries. Lineage shows the dependency of SQL queries that you write, it shows the provenance of data and it shows you downstream usage of a given table or query which is important for impact analysis. For example if you change a query and remove a column, the lineage will show you which downstream processes might brake.
To see the lineage of a table, go to a table and click on “Details” in the top menu and expand the Lineage section. Click on a block in the lineage flow to see details of that table and to go into that table.
Next to the automatically generated lineage, you can also add your own lineage by adding annotations to a table or to a Python script.
Lineage annotations
Add annotations to a table or Python script in order to add a node to the lineage graph:
- For tables: add annotations in the description field of a table (under Details)
- For scripts: add annotations in the Python script code
Example annotation for an SQL query (table):
"""
:lineage_annotation:
:lineage_ref: query_vip_customers
:lineage_ref_source_annotation: hubspot_companies
"""Example annotation for a script:
"""
:lineage_annotation:
:lineage_title: Sync data to CRM
:lineage_img: https://app.eu.peliqan.io/img/python.svg
:lineage_color: #f1eada
:lineage_ref: crm_sync_data
:lineage_ref_target_annotation: some_ref_from_a_table_or_another_script
"""Parameters in the annotation:
lineage_ref: unique id of the item (used to reference it from another node)lineage_ref_source_annotation: incoming link from a “source” node, use thelineage_refof the source node to reference itlineage_ref_target_annotation: outgoing link to a “target” node, use thelineage_refof the target node to reference itlineage_title:the title displayed in the node in the lineage graphlineage_img: the icon displayed in the node in the lineage graphlineage_color: the background color used for the node in the lineage graph