2. What are some examples of policies I should use to delegate access to Qubole for my Cloud accounts?
QDS needs access privileges to your Cloud account to do the following:
Launch and terminate machines
Create and delete security groups
Read access to data you wish to query
Read/Write access to an object store location to store Hive tables, results, and logs
The following topics provide more information:
2.1. Managing Access for Azure
Detailed Guide explains how to configure Azure resources for access by QDS. This Azure document explains how to assign access to Azure resources via Active Directory.
2.2. Managing Access for AWS
Sample IAM policies for EC2 settings and S3 policy are provided below. In addition, refer to:
Here are two sample IAM policies for EC2 settings.
Sample 1 - Simple Policy for EC2 Settings
This sample is a simpler AWS policy for EC2 settings and it does:
Support EBS Upscaling (supported by policy elements containing
Volume). Sample Policy for EBS Upscaling provides the exact policy elements required for EBS Upscaling.Support heterogeneity in clusters (supported by
SpotFleetcontaining-policy elements). AWS Considerations provides more information.Include policy elements containing
Rolethat are applicable to IAM Roles.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:AttachVolume",
"ec2:CancelSpotInstanceRequests",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteSecurityGroup",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:Describe*",
"ec2:DescribeVolumes",
"ec2:DetachVolume",
"ec2:ImportKeyPair",
"ec2:DescribeKeyPairs",
"ec2:ModifyInstanceAttribute",
"ec2:RequestSpotInstances",
"ec2:RevokeSecurityGroupIngress",
"ec2:RunInstances",
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:RequestSpotFleet",
"ec2:DescribeSpotFleetInstances",
"ec2:DescribeSpotFleetRequests",
"ec2:DescribeSpotFleetRequestHistory",
"ec2:CancelSpotFleetRequests",
"iam:PassRole",
"iam:ListRoles",
"iam:GetRole",
"iam:ListInstanceProfiles",
"iam:SimulatePrincipalPolicy"
],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": ["sts:DecodeAuthorizationMessage"],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateNetworkInterface",
"ec2:DetachNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeNetworkInterfaceAttribute",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:DeleteNetworkInterface",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:CreateServiceLinkedRole",
"iam:PutRolePolicy"
],
"Resource": ["arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot", "arn:aws:iam::*:role/aws-service-role/spotfleet.amazonaws.com/AWSServiceRoleForEC2SpotFleet"],
"Condition": {
"StringLike": {
"iam:AWSServiceName": ["spot.amazonaws.com","spotfleet.amazonaws.com"]
}
}
}
]
}
Note
You must mention your bucket path here in place of <bucketpath>. Users of an account must be allowed access
to the default S3 bucket location that is configured in Configuring your Access Settings using IAM Roles. In the policy sample, it is
represented by the policy Action element, s3:GetBucketLocation and the policy Resource element,
arn:aws:s3:::<bucketpath>.
Sample 2 - EC2 Settings Policy with Restrictive Permissions
Here is a sample with more restrictive policy permissions and it does:
Limit RunInstances, CreateTags, DeleteTags to a particular VPC (supported by
vpc/<vpc-id>).Only terminate instances which are brought up (based on the instance profile)
Only authorize SG ingress/egress for SGs created by Qubole (via tag)
Support EBS Upscaling (supported by policy elements containing
Volume). Sample Policy for EBS Upscaling provides the exact policy elements required for EBS Upscaling.Support heterogeneity in clusters (supported by
SpotFleetcontaining-policy elements). AWS Considerations provides more information.Include policy elements containing
Rolethat are applicable to IAM Roles.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "NonResourceBasedPermissions",
"Effect": "Allow",
"Action": [
"ec2:AssociateAddress",
"ec2:DisassociateAddress",
"ec2:ImportKeyPair",
"ec2:RequestSpotInstances",
"ec2:RequestSpotFleet",
"ec2:ModifySpotFleetRequest",
"ec2:CancelSpotFleetRequests",
"ec2:CancelSpotInstanceRequests",
"ec2:CreateSpotDatafeedSubscription",
"ec2:DeleteSpotDatafeedSubscription",
"ec2:Describe*",
"ec2:CreateKeyPair",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:CreateFleet",
"ec2:DeleteLaunchTemplate",
"ec2:DeleteLaunchTemplateVersions",
"ec2:CreateLaunchTemplateVersion",
"ec2:CreateLaunchTemplate",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeLaunchTemplateVersions",
"sts:DecodeAuthorizationMessage",
"iam:SimulatePrincipalPolicy"
],
"Resource": ["*"]
},
{
"Sid": "AllowInstanceActions",
"Effect": "Allow",
"Action": [
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:ModifyInstanceAttribute",
"ec2:TerminateInstances",
"ec2:AttachVolume",
"ec2:DetachVolume",
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": "arn:aws:ec2:<AWS Region>:<AWS Account ID>:instance/*",
"Condition": {
"StringLike": {
"ec2:InstanceProfile": "arn:aws:iam::<AWS Account ID>:instance-profile/<AWS Role Name>"
}
}
},
{
"Sid": "RunInstanceWithRole",
"Effect": "Allow",
"Action": [
"ec2:RunInstances",
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": [
"arn:aws:ec2:<AWS Region>:<AWS Account ID>:launch-template/*",
"arn:aws:ec2:<AWS Region>:<AWS Account ID>:instance/*"
],
"Condition": {
"StringLike": {
"ec2:InstanceProfile": "arn:aws:iam::<AWS Account ID>:instance-profile/<AWS Role Name>"
}
}
},
{
"Sid": "RunInstanceInSubnet",
"Effect": "Allow",
"Action": [
"ec2:RunInstances",
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": ["arn:aws:ec2:<AWS Region>:<AWS Account ID>:subnet/*"],
"Condition": {
"StringLike": {
"ec2:vpc": "arn:aws:ec2:<AWS Region>:<AWS Account ID>:vpc/*"
}
}
},
{
"Sid": "RunInstanceResourcePermissions",
"Effect": "Allow",
"Action": [
"ec2:RunInstances",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:AuthorizeSecurityGroupEgress"
],
"Resource": [
"arn:aws:ec2:<AWS Region>::image/*",
"arn:aws:ec2:<AWS Region>::snapshot/*",
"arn:aws:ec2:<AWS Region>:<AWS Account ID>:volume/*",
"arn:aws:ec2:<AWS Region>:<AWS Account ID>:network-interface/*",
"arn:aws:ec2:<AWS Region>:<AWS Account ID>:key-pair/*",
"arn:aws:ec2:<AWS Region>:<AWS Account ID>:security-group/*",
"arn:aws:ec2:<AWS Region>:<AWS Account ID>:launch-template/*"
]
},
{
"Sid": "SecurityGroupActions",
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:RevokeSecurityGroupEgress",
"ec2:DeleteSecurityGroup",
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": ["*"],
"Condition": {
"StringLike": {
"ec2:vpc": "arn:aws:ec2:<AWS Region>:<AWS Account ID>:vpc/*"
}
}
},
{
"Sid": "CreateAndDeleteVolumeActions",
"Effect": "Allow",
"Action": [
"ec2:CreateVolume",
"ec2:DeleteVolume",
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": "arn:aws:ec2:<AWS Region>:<AWS Account ID>:volume/*"
},
{
"Effect": "Allow",
"Action": [
"iam:CreateServiceLinkedRole",
"iam:PutRolePolicy"
],
"Resource": [
"arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot",
"arn:aws:iam::*:role/aws-service-role/spotfleet.amazonaws.com/AWSServiceRoleForEC2SpotFleet"
],
"Condition": {
"StringLike": {
"iam:AWSServiceName": [
"spot.amazonaws.com",
"spotfleet.amazonaws.com"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateNetworkInterface",
"ec2:DetachNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeNetworkInterfaceAttribute",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:DeleteNetworkInterface",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole",
"iam:GetRole"
],
"Resource": "arn:aws:iam::<AWS Account ID>:role/qubole-ec2-spot-fleet-role"
}
]
}
Note
You must mention your bucket path here in place of <bucketpath>. Users of an account must be allowed access
to the default S3 bucket location that is configured in Configuring your Access Settings using IAM Roles. In the policy sample, it is
represented by the policy Action element, s3:GetBucketLocation and the policy Resource element,
arn:aws:s3:::<bucketpath>. You need to also mention the corresponding AWS region that is represented by
<aws-region> in the above sample.
Simple AWS S3 Policy
Here is a sample IAM policy for creating an AWS S3 policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetBucketAcl",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::<bucketpath>/*",
"arn:aws:s3:::<bucketpath>/",
"arn:aws:s3:::<bucketpath>"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [ "*" ]
},
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": [ "*" ]
}
]
}
Sample AWS S3 Policy with VPC Endpoints
Here is a sample policy for whitelisting VPC endpoints to access S3 buckets.
Note
A VPC endpoint is unique for a given QDS environment. To know VPC endpoints (represented by <vpc endpoint>
in the below policy), see VPC endpoints for QDS environments.
{
"Version": "2012-10-17",
"Id": "Policy1415115909152",
"Statement": [
{
"Sid": "Grant access to Qubole VPC Endpoint",
"Principal": "*",
"Action": "s3:*",
"Effect": "Allow",
"Resource": ["arn:aws:s3:::<bucketpath>",
"arn:aws:s3:::<bucketpath>/*"],
"Condition": {
"StringEquals": {
"aws:sourceVpce": "<vpc endpoint>"
}
}
}
]
}
Sample Policy for IAM Roles
Here is a sample policy for a cross-account IAM role. See Creating a Cross-account IAM Role for QDS for more information.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:GetInstanceProfile",
"Resource": "arn:aws:iam::<arn_number>:instance-profile/<role_name>"
},
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::<arn_number>:role/<role_name>"
}
]
}
Here is a sample policy to update trust relationships of a cross-account IAM role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:: quboleawsaccountid:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "externalid"
}
}
}
]
}
2.3. Sample Basic S3 Policy to Restrict a User with Read-only Access
Here is a sample basic S3 policy restricting a user with only read-only access defined by the Action policy element.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "<Stmt1384923783000>",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::paid-qubole/*"
]
},
{
"Sid": "Stmt1384923783000",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::paid-qubole/*"
]
}
]
}
2.4. Sample Policy for Limiting Access to a VPC
Here is a sample policy for allowing QDS to limit access to a VPC.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource":
[
"arn:aws:ec2:<region>:<accountID>:subnet/*",
"arn:aws:ec2:<region>:<accountID>:network-interface/*",
"arn:aws:ec2:<region>:<accountID>:security-group/*"
],
"Condition":
{
"StringEquals":
{ "ec2:Vpc":"arn:aws:ec2:<region>:<accountID>:vpc/<vpc-ID>" }
}
},
{
"Effect": "Allow",
"Action":
[
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress"
],
"Resource":
[
"arn:aws:ec2:<region>:<accountID>:volume/*",
"arn:aws:ec2:<region>:<accountID>:instance/*",
"arn:aws:ec2:<region>:<accountID>:placement-group/*",
"arn:aws:ec2:<region>:<accountID>:security-group/*",
"arn:aws:ec2:<region>1:*:image/*",
"arn:aws:ec2:<region>:<accountID>:network-interface/*",
"arn:aws:ec2:<region>:*:key-pair/*"
]
},
{
"Effect": "Allow",
"Action":
[
"ec2:RunInstances"
],
"Resource":
[
"arn:aws:ec2:<region>:<accountID>:volume/*",
"arn:aws:ec2:<region>:<accountID>:instance/*",
"arn:aws:ec2:<region>:<accountID>:placement-group/*",
"arn:aws:ec2:<region>:*:image/*",
"arn:aws:ec2:<region>:*:key-pair/*"
]
},
{
"Effect": "Allow",
"Action":
[
"ec2:CancelSpotInstanceRequests",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:DeleteSecurityGroup",
"ec2:DeleteTags",
"ec2:Describe*",
"ec2:ImportKeyPair",
"ec2:ModifyInstanceAttribute",
"ec2:RequestSpotInstances"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": ["sts:DecodeAuthorizationMessage"],
"Resource": ["*"]
}
]
}
Sample Policy for Elastic IP Address
Here is a sample policy with EC2 settings’ to allow QDS for adding and managing Elastic IP address.
{
"Version": "2012-10-17",
"Statement":
[
{
"Effect": "Allow",
"Action":
[
"ec2:AssociateAddress",
"ec2:DescribeAddresses",
"ec2:DisassociateAddress"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": ["sts:DecodeAuthorizationMessage"],
"Resource": ["*"]
}
]
}
Sample Policy for EBS Upscaling
QDS must have the ability to perform operations on EBS volumes to use EBS Upscaling in AWS. Here is an example IAM policy that grants the required privileges.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateVolume",
"ec2:DeleteVolume",
"ec2:DescribeVolumes",
"ec2:DetachVolume",
"ec2:ModifyInstanceAttribute"
],
"Resource": [
"*"
]
}
]
}
Sample Policies for EBS Encryption
These are the two policies that you need to attach:
Providing Permissions for Spot Instances on Clusters with Custom KMS Keys Encryption
Providing Permissions to Launch Clusters with Custom KMS Keys Encryption
Providing Permissions for Spot Instances on Clusters with Custom KMS Keys Encryption
Add this policy to provide permissions to the AWSServiceRoleForEC2Spot to access the custom KMS key and use it for
encryption and it is required for Spot instances launched as part of clusters.
Note
The custom KMS encryption is only supported for the instance types described in: EBS Encryption. You must only use the supported instances described in the above referenced topic (EBS Encryption) if you require the custom KMS encryption for EBS volumes.
{
"Version": "2012-10-17",
"Id": "key-consolepolic",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-id>:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow access for Key Administrators",
"Effect": "Allow",
"Principal": {
"AWS": "<USER/ROLE-ARN>"
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:TagResource",
"kms:UntagResource",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
],
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::330183209093:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:ViaService": "ec2.us-east-1.amazonaws.com"
}
}
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::330183209093:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "<KMS_KEY_ARN>",
"Condition": {
"StringEquals": {
"kms:ViaService": [
"ec2.us-east-1.amazonaws.com",
]
},
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
}
Providing Permissions to Launch Clusters with Custom KMS Keys Encryption
Attach this policy to the Qubole primary and secondary AWS IAM Roles to successfully launch clusters after enabling encryption with custom KMS keys (SSE-C). It can also work with AWS IAM Keys.
Note
The custom KMS encryption is only supported for the instance types described in: EBS Encryption. You must only use the supported instances described in the above referenced topic (EBS Encryption) if you require the custom KMS encryption for EBS volumes.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey",
"kms:GenerateDataKeyWithoutPlaintext",
"kms:DescribeKey",
"kms:ReEncrypt*"
],
"Resource": "<KMS_KEY_ARN>",
"Condition": {
"StringEquals": {
"kms:ViaService": [
"ec2.us-east-1.amazonaws.com"
]
}
}
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"kms:RevokeGrant",
"kms:CreateGrant",
"kms:ListGrants"
],
"Resource": "<KMS_KEY_ARN>",
"Condition": {
"StringEquals": {
"kms:ViaService": [
"ec2.us-east-1.amazonaws.com"
]
},
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
}
The kms:ViaService condition allows different services to access and use the key based on the user/role in both these permission statements.
Sample Trust Relationship Policy
Here is a sample Trust Relationship policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AccountID>:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<ExternalID>"
}
}
},
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Sample Policy for a Single File Result Download
Here is a sample policy that you can refer to.
{
"Id": "Policy1551768215152",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1551762699682",
"Action": [
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::<defloc-bucket-location> and arn:aws:s3:::<defloc-bucket-location>/*",
"Principal": {
"AWS":["test-role"]
}
}
]
}
2.5. Managing Access for Oracle OCI
Configuring Oracle OCI Resources explains how to configure Oracle resources for access by QDS. Specifically, you can assign a group broad access to compartment resources with a statement such as this:
allow group <group_name> to manage all-resources in compartment <compartment_name>
To allow more fine-grained access to compartment resources, use statements such as these:
allow group <group_name> to read virtual-network-family in compartment <compartment_name>
allow group <group_name> to manage volume-family in compartment <compartment_name>
allow group <group_name> to manage object-family in compartment <compartment_name>
allow group <group_name> to manage instance-family in compartment <compartment_name>
This Oracle documentation provides more information.