Configuring a Cluster in a VPC with Public and Private Subnets (AWS)

This topic explains how to configure a cluster in an AWS VPC with public and private subnets. For more information (and for instructions on configuring a private subnet for Azure or Oracle) see Configuring a Private Subnet.

Configuring a cluster in a VPC with public and private subnets involves the following steps:

  1. Creating a VPC with Private and Public Subnets on AWS
  2. Configuring the Route Tables
  3. Creating a Security Group in the VPC
  4. Bringing up the Bastion Host in the Public Subnet
  5. Adding a Custom SSH Key to the Bastion Node (Optional)
  6. Assigning the Security Group to the Bastion Host
  7. Creating a Cluster in the VPC

See reference documentation for more information.

Creating a VPC with Private and Public Subnets on AWS

  1. Create a VPC.
  2. For the VPC, set the EnableDNSHostnames property to true.
  3. Create the required private and public subnets for the VPC.
  4. If you want to use a private subnet, then create a NAT gateway in a public subnet for that VPC.

See Working with VPCs and Subnets for the detailed information.

Note

Tunnelling with Bastion Nodes for Private Subnets in an AWS VPC provides a list of IP addresses to whitelist on https://api.qubole.com and https://us.qubole.com QDS environments.

Configuring the Route Tables

You must configure the route tables used with private and public subnets to control the routing for the subnet.

  • Configure Route Table used with a private subnet All outbound traffic (0.0.0.0/0) must go to the NAT gateway or the NAT instance that resides in the public subnet. Qubole recommends using a NAT gateway instead of a customer-setup NAT instance for high reliability. You should ensure that the Route Table contains an Amazon S3 endpoint as one of the routes.

    If you do not find an Amazon S3 endpoint, create a VPC endpoint to allow direct access to the AWS S3 object store in the region in which the VPC is in and add this VPC endpoint as an entry in the private subnet’s Route Table.

    Some AWS regions do not support NAT gateways. If the VPC is in such AWS region, create a NAT instance as described in NAT Instances. Set the following inbound and outbound rules in the NAT instance.

    Inbound

    • Open HTTP and HTTPS ports for private subnet CIDR.

    Outbound

    • Allow outgoing traffic on all ports to everywhere.
  • Configure Route Table used with a public subnet All outbound traffic (0.0.0.0/0) must go to the internet gateway. You should ensure that the Route Table contains an Amazon S3 endpoint as one of the routes.

    If you do not find an Amazon S3 endpoint, add the same VPC endpoint that was created for the private subnet in the route table of the public subnet.

Creating a Security Group in the VPC

You must create a Security Group for the bastion host on the VPC where you want to launch the cluster and set the following inbound and outbound settings:

Inbound

  • Refer to Tunnelling with Bastion Nodes for Private Subnets in an AWS VPC to get the Qubole tunnel server’s IP address. Allow SSH access (on port 22) to the Qubole tunnel server. You can open the SSH port access of the bastion node to the world or to only a few IP addresses by contacting Qubole Support.

    Note

    It is highly recommended to use a tunnel and not open SSH to the world.

    Once tunnelling is enabled on the cluster, it is automatically used for data export/import and running commands and so on as before.

    For more information on the ports that allow inbound traffic, see Understanding Cluster Network Security Characteristics (AWS).

  • Allow port 7000 access to the private subnet CIDR.

Outbound

  • Allow outbound traffic on all ports to everywhere. However, allowing outbound traffic to only a private subnet CIDR is also sufficient.
  • For bastion host’s security group:
    • Allow VPC CIDR IP on 0-65535 ports
    • Allow any additional user’s RDS on specified ports

Bringing up the Bastion Host in the Public Subnet

Warning

Qubole only supports one bastion node per Hive metastore database. So, if you have multiple Qubole accounts and use Qubole-managed Hive metastore, then you must not use the same bastion node across multiple Qubole accounts. Using the same bastion node may cause cluster startup failures and/or job/query failures. The communication between Qubole Control Plane and clusters in a private subnet occurs through the bastion node. So, reusing the bastion node can also be a single point of failure in the environment.

Even if you use a custom Hive metastore, which implies that Qubole accounts use the same Hive metastore, it is highly recommended to use a separate bastion node for each account to ensure high availability (HA) in case of a failure.

Qubole uses the bastion host for all forward/reverse communication from the cluster. You should bring up the bastion host in the public subnet of the VPC.

Important

To configure the bastion host, Qubole recommends using anyone of these two instances:

  • c5n.xlarge that comes with with 25 Gbps peak network throughput and 10.5 GB memory
  • m5.xlarge/m5a.xlarge that come with 10 Gbps peak network throughput and 16 GB memory

Qubole recommends attaching an Elastic IP to the bastion host. This avoids the public DNS/hostname of the bastion host from changing in the event where the bastion host goes down or requires a restart. If the bastion host’s public DNS/hostname changes, then you must manually edit all Qubole clusters (using the previous bastion host) to use the new bastion host’s public DNS. In addition, you might face connectivity issues from the Qubole Control Plane to clusters that are already running (using the previous bastion host) until they are restarted.

Caution

It is highly recommended to avoid using t2 instance types for configuring a bastion host.

Adding a Unique SSH Key while bringing up the Bastion Host

Note

This is a mandatory step if the QDS account is on any QDS-on-AWS environment. For more information, see Supported Qubole Endpoints on Different Cloud Providers.

Clusters support the account-level unique SSH key feature, which would enable Qubole to SSH into the bastion host in the subsequent logins through the unique public SSH key.

View the Account-level Public SSH Key describe the account API to view the account-level public SSH key.

Refresh the Account-level SSH Key Pair describes the API to refresh/rotate the SSH key pair. You must add this public SSH key to the bastion host by appending ~/.ssh/authorized_keys for an ec2-user. Whenever the public SSH key is rotated, ensure to replace the public SSH key with the rotated public SSH key in the bastion host.

Bringing up a bastion host basically involves starting an EC2 instance using either the Qubole-provided AMI or other AMIs:

Bring up the Bastion Host using the Qubole-provided AMI

Use the image in the Amazon community labeled qubole-bastion-hvm-amzn-linux (a hardware virtual machine (HVM) image) available in all AWS regions. You can use the HVM image to bring up instances from older generations of instance families (m1 and m2 instance families). In case if you find multiple images labeled qubole-bastion-hvm-amzn-linux, then pick the latest image because it supports all instance types including the newly supported instance types.

The image contains a Qubole Public Key, which you can use to start an EC2 instance for bringing up a bastion host in the public subnet while creating a cluster.

Bring up the Bastion Host using other AMIs

If you are not using Qubole’s AMI to bring up bastion node, perform the following steps:

  1. Modify the SSH configuration in /etc/ssh/sshd_config to add these two configurations:

    • Set GatewayPorts and AllowTcpForwarding to yes.
    • Set MaxSessions 1024 and MaxStartups 1024 to allow more than the default 10 parallel SSH sessions/connections.
  2. After editing the SSH configuration file, run sudo /etc/init.d/sshd restart to restart the SSH service.

    In case of an AL2 AMI, run sudo systemctl restart ssh to restart the SSH service.

Adding a Custom SSH Key to the Bastion Node

If you want to log in to the cluster, you must add a custom SSH key to the bastion node (that is brought up by Qubole-provided AMI/other AMIs).

Follow these steps to log into the bastion node through SSH:

  1. Use the SSH KeyPair that is used to launch the cluster instance, to log into the bastion node.
  2. Ensure that the machine from which you are using SSH, has access to the bastion node. Whitelisting the machine’s IP address in the bastion node’s security group guarantees the machine’s access to the bastion node. If there is no security group, then assign a security group to the bastion node as described in Assigning the Security Group to the Bastion Host.
  3. After logging into the bastion node, add the custom SSH key. This allows you to log into the bastion node and thus log in to the cluster.

For more information on how to log into clusters in Amazon EC2 and VPC, see:

Assigning the Security Group to the Bastion Host

Assign the Bastion Security Group to the bastion host.

The following figure illustrates a VPC with private and public subnets.

../../_images/VPC-PrivateSubnet.png

Note

For more restrictive outbound settings, see Creating a Security Group in the VPC.

Creating a Cluster in the VPC

After creating a VPC, create a cluster within that VPC by performing the following additional step:

  1. Create a cluster with above created VPC’s vpc-id and a private subnet ID within that VPC. QDS UI supports specifying a private subnet ID for the corresponding VPC. See Advanced configuration: Modifying EC2 Settings (AWS) for more information. The QDS UI also allows you to add a bastion host DNS.

Additional Information