Search

Connectors

Contact support

Github - Getting started in Peliqan

Github - Getting started in Peliqan

This article provides an overview to get started with the Github connector in Peliqan. Please contact support if you have any additional questions or remarks.

Contents

Connect

In peliqan, go to Connections, click on Add new. Find Github in the list and select it. Click on the Connect button. This will open Github and allow you to authorize access for Peliqan. Once that is done, you will return to Peliqan.

image

Explore & Combine

The Github connector in Peliqan is currently a “writeback”-only connector, this means it can be used in Python scripts but there are no out-of-the-box ETL pipelines that will sync data from Github into tables in the Peliqan data warehouse.

Writeback (Python scripts)

Below are examples of how to read and write from and to Github from Peliqan Python scripts (data apps).

Write file to Github (commit)

Here’s an example to commit files to Github from a script in Peliqan:

☝
Note that in order to update an existing file, you need the SHA. Use file_get() to retrieve the SHA of the file first, and include it when updating the file.
import base64, json

github_api = pq.connect('Github')

content = { "test_key": "test content" }
content_string = json.dumps(content)
base64_content = base64.b64encode(bytes(content_string, 'utf-8')).decode('utf-8')

file = {
    "path": "test_folder/test_file.json",
    "repo": "my_repo",
    "owner": "my_github_username",
    "message": "This is a commit from Peliqan script",
    "base64_content": base64_content,
    "committer_name": "John",
    "committer_email": "john@peliqan.io"
}

get_file = github_api.get('file', file)

if 'sha' in get_file:
    st.text("Updating file")
    file['sha'] = get_file['sha']
    result = github_api.update('file', file)
else:
    st.text("Adding new file")
    result = github_api.add('file', file)

st.text(result["status"])

Read file from Github

import base64

github_api = pq.connect('Github')
file = {
    "owner": "ownername",
    "repo": "reponame",
    "path": "my_script.py",
}
result = github_api.get('file', file)
content = base64.b64decode(result["content"]).decode("utf-8")
st.write(content)

Peliqan CI/CD using Github

Using the Github connector, you can build CI/CD pipelines that e.g. push updates of data models (queries) and Python scripts from Peliqan to Github, and that pull in updates from Github into Peliqan.

Example: pull Python script update from Github into Peliqan

import base64

github_api = pq.connect('Github')
file = {
    "owner": "ownername",
    "repo": "reponame",
    "path": "my_script.py",
}
result = github_api.get('file', file)
content = base64.b64decode(result["content"]).decode("utf-8")

# Update script in Peliqan
script_meta_data = pq.update_script(script_id = 123, raw_script = content)
st.json(script_meta_data)

Example: pull data model (query) update from Github into Peliqan

import base64

github_api = pq.connect('Github')
file = {
    "owner": "ownername",
    "repo": "reponame",
    "path": "my_data_model.sql",
}
result = github_api.get('file', file)
content = base64.b64decode(result["content"]).decode("utf-8")

# Update query (data model) in Peliqan
table_meta_data = pq.update_table(table_id = 123, query = content)
st.json(table_meta_data)

Example: commit Python script from Peliqan to Github

import base64, json

github_api = pq.connect('Github')

script_meta_data = pq.get_script(script_id = 853)
content = script_meta_data["raw_script"]
base64_content = base64.b64encode(bytes(content, 'utf-8')).decode('utf-8')

file = {
    "owner": "ownername",
    "repo": "reponame",
    "path": "my_script.py",
    "message": "This is a commit from a Peliqan script",
    "base64_content": base64_content,
    "committer_name": "John Doe",
    "committer_email": "john@acme.com"
}

get_file = github_api.get('file', file)
if 'sha' in get_file:
    file['sha'] = get_file['sha']
    result = github_api.update('file', file)
else:
    result = github_api.add('file', file)

Example: commit data model (query) from Peliqan to Github

import base64, json

github_api = pq.connect('Github')

query_meta_data = pq.get_table(table_id = 123)
content = query_meta_data["query"]
base64_content = base64.b64encode(bytes(content, 'utf-8')).decode('utf-8')

file = {
    "owner": "ownername",
    "repo": "reponame",
    "path": "my_datamodel.sql",
    "message": "This is a commit from a Peliqan data model",
    "base64_content": base64_content,
    "committer_name": "John Doe",
    "committer_email": "john@acme.com"
}

get_file = github_api.get('file', file)
if 'sha' in get_file:
    file['sha'] = get_file['sha']
    result = github_api.update('file', file)
else:
    result = github_api.add('file', file)