Create a Persistent Security Group in AWS

When QDS launches a cluster in AWS, QDS creates an AWS security group dynamically for the cluster. This works well in most cases, but because a QDS cluster is ephemeral (comprising a new set of AWS instances each time the cluster is launched), the security group is also temporary. When a QDS cluster needs to communicate with an outside entity such as a Kafka cluster, QDS needs a persistent security group.

Here’s how this works with a Kafka cluster.

Kafka brokers listen on port 9092, and all the Kafka brokers will be part of their own security group (we’ll call it KafkaSG). For a QDS cluster to have access to the Kafka cluster, KafkaSG needs to open port 9092 for the QDS cluster security group. So we’ll create a persistent security group for QDS called Qubole_PSG and add a rule in KafkaSG to open up port 9092 for Qubole_PSG.

Now any machine that is part of Qubole_PSG will be able to talk to the Kafka cluster. When you configure a QDS cluster for use with Kafka, you specify Qubole_PSG as the persistent security group. The cluster will become part of that security group (instead of the default security group QDS would otherwise create dynamically) and hence will be able talk to Kafka.

To create a persistent security group for use by QDS, proceed as follows:

  1. Follow the AWS instructions to create a generic security group; you don’t need to add any rules. Make a note of the name you give the security group (Qubole_PSG in our example).

  2. In the security configuration used by your external cluster (security group KafkaSG in our example) open the port on which that cluster listens for connections (port 9092 for Kafka).

  3. When prompted for the name of a persistent security group when you are creating a cluster in the QDS UI, provide the name you used in Step 1. Qubole only uses the security group name for validation. So, do not provide the security group’s ID.