Skip to content

Openflow - Hosted in AWS with Snowflake managed VPC

Goal of this tutorial is to setup openflow deployment in AWS with the VPC configuration manged by Snowflake. This will allow you to start adding runtimes that have connectors in them.

Video

Video still in development

Requirements

  • You can NOT be on a trial account. (Link)
  • Snowflake account has to be in an AWS region.(Link)

Opeflow

Navigate to openflow in the navbar. UPDATE

If you get the error 'Invalid consent request' or 'TOTP Invalid'

You will have to change your default role to a role that is not an admin role. Example default would be public. UPDATE

Launch openflow and login. UPDATE

Deployment

Once logged in lets click "create deployment". UPDATE

Click next. UPDATE

Select AWS and click next. UPDATE

This is where we will select Snowflake as the VPC deployment and click "create deployment". UPDATE

This will allow us to download the cloudformation template to be used in AWS next. UPDATE

AWS

Warning

Ensure your Snowflake region and AWS infrastructure are in the same region to avoid egress cost.

Lets get our deployment active by getting cloudformation to generate the necessary infrustructure. UPDATE

CloudFormation

Search cloudformation and click the icon. UPDATE

Click "create stack". UPDATE

Select existing template, select "upload a template file" and then upload the template we got from openflow. If you want to view what will be deployed then click "View in Infrastructure Composer" (1). Click next.

  1. UPDATE

UPDATE

Give the stack a name and then scroll all the way down the longest page ever to the next button. UPDATE

All the defaults are good here. Click the acklowledge and then next. UPDATE

Scroll down anther long page and click submit. UPDATE

This will kick off the creation process of all the resources. To see the process click "Resources". UPDATE

Here we can watch the process. UPDATE

Network Policy (Optional)

If you have a blocking network policy in place, please follow these steps

Lets copy the IP address so that we can copy it into our external access next. UPDATE

Note

Make sure you add the "/32" to the ip address.

Lets create the network rule and external access in a worksheet(1) so that it will allow openflow/aws/snowflake to talk.

  1. Worksheet
1
2
3
4
5
6
7
8
9
create or replace network rule openflow_network_rule
    mode = ingress
    type = IPV4
    value_list = (
        '<AWS IP ADDRESS>/32'
    );

-- Change the existing policy.
alter network policy <your_network_policy_name> allowed_network_rule_list = openflow_network_rule;

1
2
3
4
5
6
7
8
9
create or replace network rule openflow_network_rule
    mode = ingress
    type = IPV4
    value_list = (
        '3.213.253.254/32'
    );

-- Change the existing policy.
alter network policy my_network_pollicy allowed_network_rule_list = openflow_network_rule;

Statement executed succesfully.

Your network policy should be updated now to allow snowflake and aws to talk with eachother.

Progress

If you want to check on the progress

To check the deployment status, click on the EC2 instance. UPDATE

Right click the EC2 instance and click connect. UPDATE

Select "Connect using a private IP" and click Connect. UPDATE

Here we will enter the bash command below and see if we see the response "Should now work.".

1
journalctl -u openflow-apply-infrastructure -f -n 10000 | grep "should now work"

You'll see it should say, should now work but even not it still may take some time for Snowflake to update. UPDATE

Waiting

Now we can go back to openflow and refresh our window or use the refresh button. UPDATE

Warning

It can take longer then the bash command leads to believe. I waited 40 minutes before it worked.

Success

Now that your deployment is active we can move on to a runtime and add a connector. Please select one of the connector tutorials on the left. UPDATE