Understanding the Looker Integration with QDS

Using Qubole and Looker, organizations can implement self-service data exploration and visualization without having to rely on or overburden the Data Science team. This would also remove the need to build and maintain on-premises infrastructure. Qubole automatically provisions, manages and scales the big data infrastructure in the cloud, freeing the Data Engineering team and Data Scientists from the tedious task of managing it. Data teams now can take on the data preparation work that Data Scientists and Analysts require. Analysts can use Looker to make the data in Qubole available to everyone across an organization regardless of technical ability for data exploration, visualization, or sharing.

Looker users connect directly to Qubole using Qubole’s JDBC drivers. Once connected, analysts can leverage Looker’s modeling language, LookML to define and share a data model across the company. Looker enables each analyst to prototype new work and push to production, so business users can leverage this model to create and edit their own reports and dashboards. Effectively, Looker enables self-service for anyone to create custom reports from data processed in Qubole. The first time a query or a report is run, Qubole authenticates the Looker user and starts the big data engine (Presto), if it is not already running. After that, Qubole scales and manages the big data engine, running it only for the required time, thus saving users up to 40% in infrastructure cost.

Here are major benefits of the Looker-Qubole integration:

  • It allows users to choose their preferred Big Data engine (Presto) to connect Looker and Qubole.
  • It automatically provisions Qubole clusters when a Looker users runs a report or query.
  • It automatically scales Qubole clusters up and down, based on the number of Looker ad-hoc analyses and reports, saving cost by not requiring to have clusters running all the time.
  • It reduces data movement. Looker leverages Qubole directly and does not require the movement of data to do analysis.
  • It enables self-service analysis. Anyone in an organization can use Looker to analyze and share data processed in Qubole regardless of the technical ability.
  • It authenticates Looker connections through Qubole with a secure communication.
  • It saves the query history in Qubole to allow auditing of Looker queries.

Here is the Qubole-Looker Architecture diagram.