Skip to content

Openflow Version Control - Github

Goal of this tutorial is to show how you can connect to a github repo and load connectors that can be version controlled.

Video

Requirements

  • You can NOT be on a trial account. (Link)
  • Snowflake account has to be in an AWS region.(Link)

Github Access Token

We will need a persoanl access token to allow Openflow to work with our Git repository. First lets navigate to the token page. Basic Navigation

Click on settings. Navigation Settings

Next developer settings. Navigation Developer Settings

We'll be using a classic token. Select Classic Token

Next we'll enter in a name, the experation of the token, and select "repo" for the scope of the permissions. Select Permission

We'll copy our token, it will be used in a later Snowflake step. Copy Token

Snowflake

Lets head into snowflake and going into a workspace worksheet (1) and creating the nesseray objects for openflow and the github connection.

  1. Worksheet
If you don't have a database, schema, or warehouse yet. 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
-- Create a database to store our schemas.
create database if not exists raw;

-- Create the schema. The schema stores all objects.
create schema if not exists raw.network;

/*
    Warehouses are synonymous with the idea of compute
    resources in other systems. We will use this
    warehouse to query our integration and to load data.
*/
create warehouse if not exists openflow 
    warehouse_size = xsmall
    auto_suspend = 30
    initially_suspended = true;

Only required if your hosting openflow in Snowflake (SPCS)

Lets create the network rule and external access that will allow openflow/snowflake to talk with our github repo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
-- Create network rule for our Github account.
create or replace network rule github_network_rule
    mode = egress
    type = host_port
    value_list = (
        'api.github.com'
    );

-- Create one external access integration with all network rules.
create or replace external access integration openflow_external_access
    allowed_network_rules = (github_network_rule)
    enabled = true;

Integration OPENFLOW_EXTERNAL_ACCESS successfully created.

Openflow

Next we'll head into openflow to setup our runtime and add the connector. UPDATE

If you get the error 'Invalid consent request' or 'TOTP Invalid'

You will have to change your default role to a role that is not an admin role. Example default would be public. UPDATE

Click "Launch openflow". UPDATE

Add the runtime

We'll create a new runtime. UPDATE

We'll select our deployment, give the runtime a name, select our snowflake role and if deployed in Snowflake our external access intergration. UPDATE

Now we'll wait 5-10 minutes for our runtime to become usable. UPDATE

If you get the error 'Invalid consent request' or 'TOTP Invalid'

You will have to change your default role to a role that is not an admin role. Example default would be public. UPDATE

Once the runtime is "Active" we can click to go into it. UPDATE

Connect to Repository

First we'll head to controller settings. UPDATE

The registry clients and we'll add a new one via the plus button. UPDATE

We'll select github and click add. UPDATE

Once added we'll click the three dots on the right and select edit. UPDATE

We'll put in our github repository information as shown. I believe you can also put public repos as well. UPDATE

Next we'll drag "import from repository" on to the canvas. UPDATE

Now you'll be able to switch to the github registry and select your branch, bucket "aka folder to store into" and if you have a connector in there you'll be able to add it to the canvas. If not thats okay. UPDATE

Now we can modify or add our process groups to our reposity by right clicking and using one of the options avaliable to us from git. UPDATE