3. How should I set up networking for my Cloud accounts?

3.1. Setting Up Networking for AWS

  • In an EC2-Classic Qubole account, QDS launches clusters with a default security group that controls traffic among the cluster nodes and acts as a virtual firewall to the outside world. The port settings are described here.

  • In an EC2-VPC Qubole account, by default QDS launches clusters in the AWS default VPC for the AWS region. The default configuration is described here.

    • You can override this default behavior and launch clusters into a specific AWS VPC; use the Advanced tab on the QDS Clusters page to specify the VPC and subnet. See QDS Requirements for an AWS VPC.

    • To create an AWS VPC with public and private subnets, and launch a cluster in that VPC, follow these instructions.

QDS Requirements for an AWS VPC

If you decide to create a VPC for your QDS clusters, make sure that the VPC meets these requirements:

  • Has an internet gateway.

  • Has a route table with a rule that specifies the internet gateway as the destination of CIDR block 0.0.0.0/0 (allowing traffic between the VCN and internet).

  • Has a subnet with an ACL (Access Control List) that

    • Allows SSH access to all

    • Allows all traffic for all protocols of all port ranges to destination 0.0.0.0/0

3.2. Setting Up Networking for Azure

To create a virtual network (VNet) for your Qubole Azure VMs, navigate to Virtual Networks in the Azure portal, and create your network following this Azure documentation.

Use the Azure Settings section under the Advanced Configuration tab of the Clusters page in the QDS UI to configure your QDS clusters to use the VNet.

3.3. Setting Up Networking for Oracle OCI

To enable QDS to bring up clusters in Oracle OCI, you must have an Oracle VCN with the following characteristics:

  • Has an internet gateway.

  • Has a route table with a rule that specifies the internet gateway as the target of CIDR block 0.0.0.0/0 (allowing traffic between the VCN and internet).

  • Has subnets for each OCI Availability Domain in which you intend to launch QDS clusters:

    • The security list for the subnets must have the following rules at a minimum:

      • Stateful ingress rules, specifying each subnet’s CIDR as the source CIDR, allowing all protocols (and hence all ports).

      • A stateful ingress rule, specifying 0.0.0.0/0 as the source CIDR, allowing ssh access (TCP protocol, port 22).

      • A stateful egress rule, specifying 0.0.0.0/0 as the destination CIDR, allowing all protocols (and hence all ports).

These Oracle documents provide explanations and instructions:

Use the Advanced Settings tab of the Clusters page in the QDS UI to configure your QDS clusters to use the VCN.

To configure and use a private subnet for your Oracle OCI clusters, follow these instructions.