Azure Steps 1 - 4

Step 1 (Azure Portal)

Add an Azure Application to Launch QDS Clusters

Prerequisites:

  • You must have an Azure Portal account with an active subscription.

  • You must be a global administrator, or you must be a user with permission to register an application in Active Directory and to assign the application to a role. User settings under Azure Active Directory in the Azure Portal should look like this:

    ../../../../_images/Azure-user-settings.jpg
  • Users can register applications must be set to Yes.

  • Restrict access to Azure AD administration portal must be set to No.

    If this is not what you see, ask your Azure global administrator to make the necessary change before you proceed.

  • You must have the Owner or User Access Administrator role to be able to assign a role to your application. Check this as follows:

    ../../../../_images/Azure-permissions.jpg

Step 1a. Create an Azure Active Directory Application and Service Principal

In the Azure Portal, choose Azure Active Directory and App registrations:

../../../../_images/Azure-App-Registrations.jpg

Choose New application registration:

../../../../_images/Azure-New-application-registration.jpg

Provide a name and URL for the application:

../../../../_images/Azure-Create-application.jpg

Click Create; this creates the application and shows its ID and other details.

Step 1b: Copy the Application ID (also known as Client ID), Authentication Key, and Tenant ID

  • Application ID: This is the application or client ID. You need this ID to set up your Qubole account (Client ID in Step 5b):

    ../../../../_images/Azure-Application-ID.jpg
  • Authentication Key: Click Settings for the application:

    ../../../../_images/Azure-Authentication-Key-1.jpg

    Create the authentication key:

    ../../../../_images/Azure-Authentication-Key-2.jpg

    Provide a description and duration (EXPIRES) and click Save.

    ../../../../_images/Azure-Authentication-Key-3.jpg

    Note

    Make sure you copy the key value before proceeding. You will need to enter this into the QDS UI in Step 5b (Compute Client Secret).

  • Tenant ID: Copy the Tenant ID or Directory ID for your Azure Active Directory tenant. Choose Properties from the Azure Active Directory menu:

    ../../../../_images/Azure-Tenant-ID-1.jpg

    Use the click-to-copy button at the right of the Directory ID field to copy the ID:

    ../../../../_images/Azure-Tenant-ID-2.jpg

Step 1c: Assign the Application to a Role

Assign the Contributor IAM role to the application you created in Step 1a.

Prerequisite: Remember you must have the Owner or User Access Administrator role as explained at the beginning of this document.

Choose or create the Resource Group you want Qubole to use:

../../../../_images/Azure-RG.png

Choose Access control (IAM) and + Add:

../../../../_images/Azure-RG-IAM.png

Choose the Contributor role and select the users or groups you want to assign access to.

Click Save to assign the role.

Note

If the Contributor role does not meet the needs of your organization, you can modify or replace it.

You have created the application you need to launch QDS clusters. Now you need to enable programmatic deployment of QDS.

Step 2 (Azure Portal)

Enable Programmatic Deployment of QDS

Search for marketplace, and then in the marketplace search for Qubole. You should see two entries:

../../../../_images/Azure-Marketplace-2.jpg

Select each entry for Qubole Data Service and do the following for each:

  • Scroll down and click:

    ../../../../_images/Azure-want-to-deploy.jpg
  • Choose your subscription:

    ../../../../_images/Azure-Configure-Programmatic-Deployment-1.jpg ../../../../_images/Azure-Configure-Programmatic-Deployment-2.jpg
  • Click Enable and Save.

Once you have enabled both Qubole Data Service items, you have enabled QDS deployment on Azure. Now proceed to add a virtual network (VNet).

Step 3: (Azure Portal)

Create a Virtual Network

Navigate to Virtual Networks (search if you need to) and click + Add:

../../../../_images/Azure-Create-virtual-network.jpg

Click Create.

You have created a virtual network for your QDS virtual machines (VMs). Now you need to create storage.

Step 4: (Azure Portal)

Create a Blob or Azure Data Lake Storage Account

Note

Choose EITHER Step 4a for Blob storage OR Step 4b for Azure Data Lake (ADL)

Step 4a. Configure Blob Storage

  • Create the storage account:

    1. Navigate to Storage accounts in the Azure portal and click + Add.

    2. On the Create storage account screen, fill in the starred fields or let them default. The storage account must be a general-purpose account and the deployment model must be Resource Manager (these are the defaults). You can choose Standard or Premium storage. Premium storage uses solid-state disks (SSDs); this option restricts your choice among instance (virtual machine) types but provides better performance. Accoount Kind should be BlobStorage:

      ../../../../_images/Azure-Create-storage-account.jpg
    3. Click Next: Advanced >

    4. Toggle the value of Secure transfer required to Disabled

    5. Click Review + create.

  • Create a dedicated storage container for Qubole logs and results:

    1. Navigate to Storage accounts and select the account you have just created.
    2. Select Blobs.
    3. Choose + Container.
    4. Give the new container a name.
    5. Click OK.
  • Copy the Storage Access keys:

    Choose Access Keys for this storage account:

../../../../_images/Azure-Access-Keys.jpg

Note

Make a note of the access keys. You will need them to configure your Qubole account (Step 5 (QDS): Configure QDS Account Settings).

If you are using Blob storage, your Azure set-up is complete. Continue with Step 5 to start using QDS.

Step 4b. Configure Azure Data Lake Storage

  • Choose + Create a Resource and then choose Storage and Data Lake Storage Gen1:

    ../../../../_images/Azure-New-Data-Lake-Store-Gen-1.jpg
  • Provide the information for the starred fields and click Create.

  • Configure permissions to allow the application you created in Step 1a to use the Data Lake Store you have just created, as shown here. (If you don’t see the new Data Lake Store in the Data Explorer immediately, wait a few moments to allow Azure to finish creating the store, then click Refresh.)

    Allow read, write, and execute permissions and allow all sub-folders to inherit them:

    ../../../../_images/Azure-Data-Lake-permissions.jpg
Click OK.

You have created your Data Lake Store and your Azure set-up is complete. Continue with Step 5 to start using QDS.