Configuring your Access Settings using IAM Roles

Watch video.

Note

The words, Qubole and qubole are reserved by Qubole. So, do not use them in any AWS configuration. For more information, see Qubole Cluster EC2 Tags (AWS).

Under Account Settings, configure the Account Details and Notifications sections as described under Configuring your Account Settings in Configuring your Access Settings using IAM Keys, then follow the steps in Configuring Access Settings for IAM Roles.

About IAM Roles in QDS

  • IAM role is an AWS recommended practice and provides a very secure experience as compared to IAM Keys for accessing AWS resources.

  • You cannot override default AWS settings in an IAM-role-based account. Qubole plans to add support for overriding default AWS settings shortly.

  • Cross-account IAM roles can be configured only at the Qubole account level and used for:

    • All Storage access

    • All Compute access. This includes all clusters associated with the account.

  • Qubole now supports configuring dual IAM roles that is more secure as the configuration provides only one role the permission to access the data in S3 buckets at a cluster level. This does not require a cross-account role to access the S3 buckets. See Creating Dual IAM Roles for your Account for more information.

  • QDS currently supports the AWS IAM-role based authentication on cluster types (that Qubole supports).

  • QDS supports all type of commands with IAM Roles.

Note

Currently, only shell commands are supported on Airflow clusters with IAM keys and IAM roles.

Configuring Access Settings for IAM Roles

Access Mode Type - To configure an IAM role, select IAM Role. Once you select it, the IAM role access mode settings are displayed as shown in the following figure.

../../../../_images/AccessSettings-IAMRoles.png

Once an IAM role is configured, the account uses role credentials instead of access keys to interact with AWS resources.

The following text fields are associated with an IAM role:

  • Qubole AWS Account ID - The Qubole AWS account ID as shown in the above figure. It is used while creating an IAM role using an AWS console.

  • External ID - A unique ID that is generated when a new account is created. It is used while creating an IAM role using an AWS console.

  • The Role ARN - A text field in which the role ARN value is filled. Creating a Cross-account IAM Role for QDS describes how to create an IAM role during which a Role ARN is generated. Enter the value in Role ARN text field.

See Creating Dual IAM Roles for your Account for more information on configuring dual IAM roles.

Storage Settings with IAM Roles

  • Default location (for any data created) - The location where logs and output data, and so on are stored. Qubole recommends you change the Default Location.

    Warning

    To avoid adverse effects on the command latency and performance, specify a separate Amazon S3 bucket for each QDS account, avoid providing long prefixes, and sharing the same Amazon S3 bucket for all accounts.

Click Save after making changes. Click Reset to change any setting.

See Manage Roles for information on how permissions to access features are granted and restricted.

Compute Settings with IAM Roles

The options in Compute Settings are:

  • Persistent Security Group is not set by default. When an account security group is set, it is attached to all clusters by default unless it is overridden in the cluster settings.

    You can improve the security of a cluster by authorizing Qubole to generate a unique SSH key every time a cluster is started; create a support ticket to enable this capability. A unique SSH key is generated by default in https://in.qubole.com , https://us.qubole.com, and https://wellness.qubole.com environments.

    Once the SSH key is enabled, QDS uses the unique SSH key to interact with the cluster. If you want use this capability to control QDS access to a Bastion node communicating with a cluster running on a private subnet, you can use the account-level key that you must authorize on the Bastion node.

    To get the account-level key, use this API or navigate to the cluster’s Advanced Configuration, the account-level SSH key is displayed in EC2 Settings as described in Advanced configuration: Modifying EC2 Settings (AWS).

    Qubole only uses the security group name for validation. So, do not provide the security group’s ID.

  • Push Compute Settings to all clusters is an option that you can select if you want to push compute settings to all clusters in an account. It is enabled by default for all new users and users, who have entered invalid credentials.

Click Save to apply your changes or Reset to cancel them.