Create a Persistent Security Group in AWS¶
When QDS launches a cluster in AWS, QDS creates an AWS security group dynamically for the cluster. This works well in most cases, but because a QDS cluster is ephemeral (comprising a new set of AWS instances each time the cluster is launched), the security group is also temporary. When a QDS cluster needs to communicate with an outside entity such as a Kafka cluster, QDS needs a persistent security group.
Here’s how this works with a Kafka cluster.
Kafka brokers listen on port 9092, and all the Kafka brokers will be part of their own security group (we’ll call it
For a QDS cluster to have access to the Kafka cluster,
KafkaSG needs to open port 9092 for the QDS cluster security group.
So we’ll create a persistent security group for QDS called
Qubole_PSG and add a rule in
KafkaSG to open up port 9092 for
Now any machine that is part of
Qubole_PSG will be able to talk to the Kafka cluster. When you configure a QDS cluster for
use with Kafka, you specify
Qubole_PSG as the persistent security group. The cluster will become part of that security
group (instead of the default security group QDS would otherwise create dynamically) and hence will be able talk to Kafka.
To create a persistent security group for use by QDS, proceed as follows:
- Follow the
to create a generic security group; you don’t need to add any rules. Make a note of the name you give the security group
Qubole_PSGin our example).
- In the security configuration used by your external cluster (security group
KafkaSGin our example) open the port on which that cluster listens for connections (port 9092 for Kafka).
- When prompted for the name of a persistent security group when you are creating a cluster in the QDS UI, provide the name you used in Step 1. Qubole only uses the security group name for validation. So, do not provide the security group’s ID.