Azure Steps 1 - 4¶
Step 1 (Azure Portal)¶
Add an Azure Application to Launch QDS Clusters¶
You must have an Azure Portal account with an active subscription.
You must be a global administrator, or you must be a user with permission to register an application in Active Directory and to assign the application to a role. User settings under Azure Active Directory in the Azure Portal should look like this:
Users can register applications must be set to Yes.
Restrict access to Azure AD administration portal must be set to No.
If this is not what you see, ask your Azure global administrator to make the necessary change before you proceed.
Step 1a. Create an Azure Active Directory Application and Service Principal¶
In the Azure Portal, choose Azure Active Directory and App registrations:
Choose New application registration:
Provide a name and URL for the application:
Click Create; this creates the application and shows its ID and other details.
Step 1b: Copy the Application ID (also known as Client ID), Authentication Key, and Tenant ID¶
Application ID: This is the application or client ID. You need this ID to set up your Qubole account (Client ID in Step 5b):
Authentication Key: Click Settings for the application:
Create the authentication key:
Provide a description and duration (EXPIRES) and click Save.
Make sure you copy the key value before proceeding. You will need to enter this into the QDS UI in Step 5b (Compute Client Secret).
Tenant ID: Copy the Tenant ID or Directory ID for your Azure Active Directory tenant. Choose Properties from the Azure Active Directory menu:
Use the click-to-copy button at the right of the Directory ID field to copy the ID:
Step 1c: Assign the Application to a Role¶
Assign the Contributor IAM role to the application you created in Step 1a.
Choose or create the Resource Group you want Qubole to use:
Choose Access control (IAM) and + Add:
Choose the Contributor role and select the users or groups you want to assign access to.
Click Save to assign the role.
If the Contributor role does not meet the needs of your organization, you can modify or replace it.
You have created the application you need to launch QDS clusters. Now you need to enable programmatic deployment of QDS.
Step 2 (Azure Portal)¶
Enable Programmatic Deployment of QDS¶
Search for marketplace, and then in the marketplace search for Qubole. You should see two entries:
Select each entry for Qubole Data Service and do the following for each:
Scroll down and click:
Choose your subscription:
Click Enable and Save.
Once you have enabled both Qubole Data Service items, you have enabled QDS deployment on Azure. Now proceed to add a virtual network (VNet).
Step 3: (Azure Portal)¶
Create a Virtual Network¶
Navigate to Virtual Networks (search if you need to) and click + Add:
You have created a virtual network for your QDS virtual machines (VMs). Now you need to create storage.
Step 4: (Azure Portal)¶
Create a Blob or Azure Data Lake Storage Account¶
Choose EITHER Step 4a for Blob storage OR Step 4b for Azure Data Lake (ADL)
Step 4a. Configure Blob Storage¶
Create the storage account:
Navigate to Storage accounts in the Azure portal and click + Add.
On the Create storage account screen, fill in the starred fields or let them default. The storage account must be a general-purpose account and the deployment model must be Resource Manager (these are the defaults). You can choose Standard or Premium storage. Premium storage uses solid-state disks (SSDs); this option restricts your choice among instance (virtual machine) types but provides better performance. Accoount Kind should be BlobStorage:
Click Next: Advanced >
Toggle the value of Secure transfer required to Disabled
Click Review + create.
Create a dedicated storage container for Qubole logs and results:
- Navigate to Storage accounts and select the account you have just created.
- Select Blobs.
- Choose + Container.
- Give the new container a name.
- Click OK.
Copy the Storage Access keys:
Choose Access Keys for this storage account:
Make a note of the access keys. You will need them to configure your Qubole account (Step 5 (QDS): Configure QDS Account Settings).
If you are using Blob storage, your Azure set-up is complete. Continue with Step 5 to start using QDS.
Step 4b. Configure Azure Data Lake Storage¶
Choose + Create a Resource and then choose Storage and Data Lake Storage Gen1:
Provide the information for the starred fields and click Create.
Configure permissions to allow the application you created in Step 1a to use the Data Lake Store you have just created, as shown here. (If you don’t see the new Data Lake Store in the Data Explorer immediately, wait a few moments to allow Azure to finish creating the store, then click Refresh.)
Allow read, write, and execute permissions and allow all sub-folders to inherit them:
You have created your Data Lake Store and your Azure set-up is complete. Continue with Step 5 to start using QDS.