2. What are some examples of policies I should use to delegate access to Qubole for my Cloud accounts?

QDS needs access privileges to your Cloud account to do the following:

  • Launch and terminate machines
  • Create and delete security groups
  • Read access to data you wish to query
  • Read/Write access to an object store location to store Hive tables, results, and logs

The following topics provide more information:

2.1. Managing Access for Azure

Detailed Guide explains how to configure Azure resources for access by QDS. This Azure document explains how to assign access to Azure resources via Active Directory.

2.2. Managing Access for AWS

Sample IAM policies for EC2 settings and S3 policy are provided below. In addition, refer to:

Here are two sample IAM policies for EC2 settings.

Sample 1 - Simple Policy for EC2 Settings

This sample is a simpler AWS policy for EC2 settings and it does:

  • Support EBS Upscaling (supported by policy elements containing Volume). Sample Policy for EBS Upscaling provides the exact policy elements required for EBS Upscaling.
  • Support heterogeneity in clusters (supported by SpotFleet containing-policy elements). AWS Considerations provides more information.
  • Include policy elements containing Role that are applicable to IAM Roles.
{
   "Version": "2012-10-17",
   "Statement": [
    {
        "Effect": "Allow",
        "Action": [
                    "ec2:AuthorizeSecurityGroupEgress",
                    "ec2:AuthorizeSecurityGroupIngress",
                    "ec2:AttachVolume",
                    "ec2:CancelSpotInstanceRequests",
                    "ec2:CreateSecurityGroup",
                    "ec2:CreateTags",
                    "ec2:CreateVolume",
                    "ec2:DeleteSecurityGroup",
                    "ec2:DeleteTags",
                    "ec2:DeleteVolume",
                    "ec2:Describe*",
                    "ec2:DescribeVolumes",
                    "ec2:DetachVolume",
                    "ec2:ImportKeyPair",
                    "ec2:DescribeKeyPairs",
                    "ec2:ModifyInstanceAttribute",
                    "ec2:RequestSpotInstances",
                    "ec2:RevokeSecurityGroupIngress",
                    "ec2:RunInstances",
                    "ec2:StartInstances",
                    "ec2:StopInstances",
                    "ec2:TerminateInstances",
                    "ec2:RequestSpotFleet",
                    "ec2:DescribeSpotFleetInstances",
                    "ec2:DescribeSpotFleetRequests",
                    "ec2:DescribeSpotFleetRequestHistory",
                    "ec2:CancelSpotFleetRequests",
                    "iam:PassRole",
                    "iam:ListRoles",
                    "iam:GetRole",
                    "iam:ListInstanceProfiles"
                   ],
        "Resource": ["*"]
    },
    {
        "Effect": "Allow",
        "Action": ["sts:DecodeAuthorizationMessage"],
        "Resource": ["*"]
    },
    {
        "Effect": "Allow",
        "Action": [
              "iam:CreateServiceLinkedRole",
              "iam:PutRolePolicy"
        ],
        "Resource": ["arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot", "arn:aws:iam::*:role/aws-service-role/spotfleet.amazonaws.com/AWSServiceRoleForEC2SpotFleet"],
           "Condition": {
              "StringLike": {
                  "iam:AWSServiceName": ["spot.amazonaws.com","spotfleet.amazonaws.com"]
            }
          }
    }
    ]
    }

Note

You must mention your bucket path here in place of <bucketpath>. Users of an account must be allowed access to the default S3 bucket location that is configured in Configuring your Access Settings using IAM Roles. In the policy sample, it is represented by the policy Action element, s3:GetBucketLocation and the policy Resource element, arn:aws:s3:::<bucketpath>.

Sample 2 - EC2 Settings Policy with Restrictive Permissions

Here is a sample with more restrictive policy permissions and it does:

  • Limit RunInstances, CreateTags, DeleteTags to a particular VPC (supported by vpc/<vpc-id>).

  • Only terminate instances which are brought up (based on the instance profile)

  • Only authorize SG ingress/egress for SGs created by Qubole (via tag)

  • Support EBS Upscaling (supported by policy elements containing Volume). Sample Policy for EBS Upscaling provides the exact policy elements required for EBS Upscaling.

  • Support heterogeneity in clusters (supported by SpotFleet containing-policy elements). AWS Considerations provides more information.

  • Include policy elements containing Role that are applicable to IAM Roles.

    { "Version": "2012-07-01",
         "Statement":
         [
         { "Sid": "NonResourceBasedPermissions",
           "Effect": "Allow",
           "Action": [
             "ec2:AssociateAddress",
             "ec2:DisassociateAddress",
             "ec2:ImportKeyPair",
             "ec2:RequestSpotInstances",
             "ec2:RequestSpotFleet",
             "ec2:ModifySpotFleetRequest",
             "ec2:CancelSpotFleetRequests",
             "ec2:CancelSpotInstanceRequests",
             "ec2:CreateSpotDatafeedSubscription",
             "ec2:DeleteSpotDatafeedSubscription",
             "ec2:Describe*",
             "ec2:CreateKeyPair",
             "ec2:CreateSecurityGroup",
             "ec2:CreateTags",
             "sts:DecodeAuthorizationMessage" ],
           "Resource": [ "*" ]
         },
         { "Sid": "AllowInstanceActions",
           "Effect": "Allow",
           "Action": [
             "ec2:StartInstances",
             "ec2:StopInstances",
             "ec2:ModifyInstanceAttribute",
             "ec2:TerminateInstances",
             "ec2:AttachVolume",
             "ec2:DetachVolume",
             "ec2:CreateTags",
             "ec2:DeleteTags" ],
           "Resource": [ "arn:aws:ec2:<AWS Region>:<AWS Account ID>:instance/*" ],
           "Condition": {
             "StringEquals": {
                  "ec2:InstanceProfile": "arn:aws:iam::<AWS Account ID>:instance-profile/<AWS Role Name>"
             }
           }
         },
         { "Sid": "RunInstanceWithRole",
           "Effect": "Allow",
           "Action": [
             "ec2:RunInstances",
             "ec2:CreateTags",
             "ec2:DeleteTags" ],
         "Resource": [ "arn:aws:ec2:<AWS Region>:<AWS Account ID>:instance/*" ],
         "Condition": {
             "StringEquals": {
                 "ec2:InstanceProfile": "arn:aws:iam::<AWS Account ID>:instance-profile/<AWS Role Name>"
             }
           }
         },
         { "Sid": "RunInstanceInSubnet",
           "Effect": "Allow",
           "Action": [
             "ec2:RunInstances",
             "ec2:CreateTags",
             "ec2:DeleteTags" ],
         "Resource": [ "arn:aws:ec2:<AWS Region>:<AWS Account ID>:subnet/*" ],
         "Condition": {
             "StringEquals": {
                  "ec2:vpc": "arn:aws:ec2:<AWS Region>:<AWS Account ID>:vpc/<VPC ID>"
             }
           }
         },
         { "Sid": "RunInstanceResourcePermissions",
           "Effect": "Allow",
           "Action": [
             "ec2:RunInstances",
             "ec2:CreateTags",
             "ec2:DeleteTags" ],
         "Resource": [
             "arn:aws:ec2:<AWS Region>::image/*",
             "arn:aws:ec2:<AWS Region>::snapshot/*",
             "arn:aws:ec2:<AWS Region>:<AWS Account ID>:volume/*",
             "arn:aws:ec2:<AWS Region>:<AWS Account ID>:network-interface/*",
             "arn:aws:ec2:<AWS Region>:<AWS Account ID>:key-pair/*",
             "arn:aws:ec2:<AWS Region>:<AWS Account ID>:security-group/*" ]
         },
         { "Sid": "SecurityGroupActions",
           "Effect": "Allow",
           "Action": [
             "ec2:AuthorizeSecurityGroupEgress",
             "ec2:AuthorizeSecurityGroupIngress",
             "ec2:RevokeSecurityGroupIngress",
             "ec2:RevokeSecurityGroupEgress",
             "ec2:DeleteSecurityGroup",
             "ec2:CreateTags",
             "ec2:DeleteTags" ],
         "Resource": [ "*" ],
         "Condition": {
             "StringEquals": {
                 "ec2:vpc": "arn:aws:ec2:<AWS Region>:<AWS Account ID>:vpc/<VPC ID>"
             }
           }
         },
         { "Sid": "CreateAndDeleteVolumeActions",
           "Effect": "Allow",
           "Action": [
             "ec2:CreateVolume",
             "ec2:DeleteVolume",
             "ec2:CreateTags",
             "ec2:DeleteTags" ],
         "Resource": [
             "arn:aws:ec2:<AWS Region>:<AWS Account ID>:volume/*" ]
         },
         {
         "Effect": "Allow",
         "Action": [
            "iam:CreateServiceLinkedRole",
            "iam:PutRolePolicy"
         ],
         "Resource": ["arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot", "arn:aws:iam::*:role/aws-service-role/spotfleet.amazonaws.com/AWSServiceRoleForEC2SpotFleet"],
         "Condition": {
            "StringLike": {
                "iam:AWSServiceName": ["spot.amazonaws.com","spotfleet.amazonaws.com"]
            }
         }
         },
         {
         "Effect": "Allow",
         "Action": [
             "iam:PassRole",
             "iam:GetRole"
         ],
         "Resource": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/qubole-ec2-spot-fleet-role"
         }
       ]
    }
    

Note

You must mention your bucket path here in place of <bucketpath>. Users of an account must be allowed access to the default S3 bucket location that is configured in Configuring your Access Settings using IAM Roles. In the policy sample, it is represented by the policy Action element, s3:GetBucketLocation and the policy Resource element, arn:aws:s3:::<bucketpath>. You need to also mention the corresponding AWS region that is represented by <aws-region> in the above sample.

Simple AWS S3 Policy

Here is a sample IAM policy for creating an AWS S3 policy.

{
 "Version": "2012-10-17",
 "Statement": [
  {
    "Effect": "Allow",
    "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetBucketAcl",
                "s3:ListBucket",
                "s3:GetBucketLocation"
              ],
    "Resource": [
                 "arn:aws:s3:::<bucketpath>/*",
                 "arn:aws:s3:::<bucketpath>/",
                 "arn:aws:s3:::<bucketpath>"
                 ]
  },
  {
    "Effect": "Allow",
    "Action": [
               "s3:GetObject",
               "s3:ListBucket",
               "s3:GetBucketLocation"
              ],
    "Resource": [ "*" ]
  },
  {
    "Effect": "Allow",
    "Action": [
               "s3:ListAllMyBuckets"
              ],
    "Resource": [ "*" ]
  }
]
}

Sample AWS S3 Policy with VPC Endpoints

Here is a sample policy for whitelisting VPC endpoints to access S3 buckets.

Note

A VPC endpoint is unique for a given QDS environment. To know VPC endpoints (represented by <vpc endpoint> in the below policy), see VPC endpoints for QDS environments.

{
  "Version": "2012-10-17",
  "Id": "Policy1415115909152",
  "Statement": [
    {
      "Sid": "Grant access to Qubole VPC Endpoint",
      "Principal": "*",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::<bucketpath>",
                   "arn:aws:s3:::<bucketpath>/*"],
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "<vpc endpoint>"
        }
      }
    }
  ]
}

Sample Policy for IAM Roles

Here is a sample policy for a cross-account IAM role. See Creating a Cross-account IAM Role for QDS for more information.

{
 "Version": "2012-10-17",
 "Statement": [
               {
                 "Effect": "Allow",
                 "Action": "iam:GetInstanceProfile",
                 "Resource": "arn:aws:iam::<arn_number>:instance-profile/<role_name>"
               },
               {
                 "Effect": "Allow",
                 "Action": "iam:PassRole",
                 "Resource": "arn:aws:iam::<arn_number>:role/<role_name>"
               }
              ]
}

Here is a sample policy to update trust relationships of a cross-account IAM role.

{
"Version": "2012-10-17",
"Statement": [
             {
              "Effect": "Allow",
              "Principal": {
                            "Service": "ec2.amazonaws.com"
                           },
              "Action": "sts:AssumeRole"
              },
              {
              "Effect": "Allow",
              "Principal": {
                            "AWS": "arn:aws:iam:: quboleawsaccountid:root"
                           },
              "Action": "sts:AssumeRole",
              "Condition": {
                             "StringEquals": {
                                               "sts:ExternalId": "externalid"
                                             }
                           }
              }
              ]
}

2.3. Sample Basic S3 Policy to Restrict a User with Read-only Access

Here is a sample basic S3 policy restricting a user with only read-only access defined by the Action policy element.

 {
   "Version": "2012-10-17",
   "Statement": [
    {
    "Sid": "<Stmt1384923783000>",
    "Action": [
    "s3:ListBucket",
    "s3:GetBucketLocation"
    ],
    "Effect": "Allow",
    "Resource": [
    "arn:aws:s3:::paid-qubole/*"
    ]
    },
    {
    "Sid": "Stmt1384923783000",
    "Action": "s3:*",
    "Effect": "Allow",
    "Resource": [
    "arn:aws:s3:::paid-qubole/*"
    ]
    }
    ]
}

2.4. Sample Policy for Limiting Access to a VPC

Here is a sample policy for allowing QDS to limit access to a VPC.

{
  "Version": "2012-10-17",
  "Statement":  [{
  "Effect": "Allow",
  "Action": "ec2:RunInstances",
  "Resource":
  [
    "arn:aws:ec2:<region>:<accountID>:subnet/*",
    "arn:aws:ec2:<region>:<accountID>:network-interface/*",
    "arn:aws:ec2:<region>:<accountID>:security-group/*"
  ],
  "Condition":
  {
    "StringEquals":
      { "ec2:Vpc":"arn:aws:ec2:<region>:<accountID>:vpc/<vpc-ID>" }
  }
  },
  {
  "Effect": "Allow",
  "Action":
     [
       "ec2:StartInstances",
       "ec2:StopInstances",
       "ec2:TerminateInstances",
       "ec2:AuthorizeSecurityGroupEgress",
       "ec2:AuthorizeSecurityGroupIngress"
     ],
  "Resource":
     [
       "arn:aws:ec2:<region>:<accountID>:volume/*",
       "arn:aws:ec2:<region>:<accountID>:instance/*",
       "arn:aws:ec2:<region>:<accountID>:placement-group/*",
       "arn:aws:ec2:<region>:<accountID>:security-group/*",
       "arn:aws:ec2:<region>1:*:image/*",
       "arn:aws:ec2:<region>:<accountID>:network-interface/*",
       "arn:aws:ec2:<region>:*:key-pair/*"
     ]
  },
  {
  "Effect": "Allow",
  "Action":
    [
      "ec2:RunInstances"
    ],
  "Resource":
    [
      "arn:aws:ec2:<region>:<accountID>:volume/*",
      "arn:aws:ec2:<region>:<accountID>:instance/*",
      "arn:aws:ec2:<region>:<accountID>:placement-group/*",
      "arn:aws:ec2:<region>:*:image/*",
      "arn:aws:ec2:<region>:*:key-pair/*"
    ]
  },
  {
   "Effect": "Allow",
   "Action":
    [
      "ec2:CancelSpotInstanceRequests",
      "ec2:CreateSecurityGroup",
      "ec2:CreateTags",
      "ec2:DeleteSecurityGroup",
      "ec2:DeleteTags",
      "ec2:Describe*",
      "ec2:ImportKeyPair",
      "ec2:ModifyInstanceAttribute",
      "ec2:RequestSpotInstances"
    ],
   "Resource": "*"
  },
  {
   "Effect": "Allow",
   "Action": ["sts:DecodeAuthorizationMessage"],
   "Resource": ["*"]
  }
  ]
}

Sample Policy for Elastic IP Address

Here is a sample policy with EC2 settings’ to allow QDS for adding and managing Elastic IP address.

{
  "Version": "2012-10-17",
  "Statement":
  [
   {
    "Effect": "Allow",
    "Action":
    [
      "ec2:AssociateAddress",
      "ec2:DescribeAddresses",
      "ec2:DisassociateAddress"
    ],
    "Resource": [
                 "*"
                 ]
   },
   {
     "Effect": "Allow",
     "Action": ["sts:DecodeAuthorizationMessage"],
     "Resource": ["*"]
   }
  ]
}

Sample Policy for EBS Upscaling

QDS must have the ability to perform operations on EBS volumes to use EBS Upscaling in AWS. Here is an example IAM policy that grants the required privileges.

{
 "Version": "2012-10-17",
 "Statement": [
     {
      "Effect": "Allow",
      "Action":   [
                    "ec2:AttachVolume",
                    "ec2:CreateVolume",
                    "ec2:DeleteVolume",
                    "ec2:DescribeVolumes",
                    "ec2:DetachVolume",
                    "ec2:ModifyInstanceAttribute"
                   ],
      "Resource":  [
                     "*"
                   ]
     }
   ]
}

Sample Policy for EBS Encryption

Attach this policy to the Qubole primary and secondary AWS IAM Roles to successfully launch clusters after enabling EBS encryption. It can also work with AWS IAM Keys.

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "AllowCMKGrantForEBSEncryption",
        "Effect": "Allow",
        "Action": "kms:CreateGrant",
        "Resource": "arn:aws:kms:*:057775362311:key/<KMS-Key>",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    },
    {
        "Sid": "AllowCMKAccessForEBSEncryption",
        "Effect": "Allow",
        "Action": [
            "kms:Decrypt",
            "kms:ReEncryptTo",
            "kms:GenerateDataKeyWithoutPlaintext",
            "kms:DescribeKey",
            "kms:ReEncryptFrom"
        ],
        "Resource": "arn:aws:kms:*:057775362311:key/<KMS-Key>"
    }
    ]
}

Sample Trust Relationship Policy

Here is a sample Trust Relationship policy.

{
 "Version": "2012-10-17",
 "Statement": [
    {
    "Effect": "Allow",
    "Principal": {
    "AWS": "arn:aws:iam::<AccountID>:root"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
    "StringEquals": {
    "sts:ExternalId": "<ExternalID>"
             }
           }
 },
 {
   "Effect": "Allow",
   "Principal": {
             "Service": "ec2.amazonaws.com"
            },
 "Action": "sts:AssumeRole"
   }
  ]
 }

Sample Policy for a Single File Result Download

Here is a sample policy that you can refer to.

{
  "Id": "Policy1551768215152",
  "Version": "2012-10-17",
  "Statement": [
  {
   "Sid": "Stmt1551762699682",
   "Action": [
              "s3:AbortMultipartUpload",
              "s3:DeleteObject",
              "s3:GetBucketLocation",
              "s3:GetObject",
              "s3:ListBucket",
              "s3:ListMultipartUploadParts"
              ],
   "Effect": "Allow",
   "Resource": "arn:aws:s3:::<defloc-bucket-location> and arn:aws:s3:::<defloc-bucket-location>/*",
   "Principal": {
   "AWS":["test-role"]
       }
   }
  ]
}

2.5. Managing Access for Oracle OCI

Configuring Oracle OCI Resources explains how to configure Oracle resources for access by QDS. Specifically, you can assign a group broad access to compartment resources with a statement such as this:

allow group <group_name> to manage all-resources in compartment <compartment_name>

To allow more fine-grained access to compartment resources, use statements such as these:

allow group <group_name> to read virtual-network-family in compartment <compartment_name>
allow group <group_name> to manage volume-family in compartment <compartment_name>
allow group <group_name> to manage object-family in compartment <compartment_name>
allow group <group_name> to manage instance-family in compartment <compartment_name>

This Oracle documentation provides more information.