Using Macros in Qubole

Macros can be used with commands and job schedules on QDS. However, Qubole only supports macros in the Scheduler UI and plans to support for UI command editor in the near future.

Macros can be accessed or defined using the Javascript language. Only assignment statements are valid. Loops, function definitions, and all other language constructs are not supported. Assignment statements can use all operators and functions defined for the objects used in the statements. Defined macros can be used in subsequent statements.

Javascript Language and Modules

The following Javascript libraries are available.

Library Description Link to Documentation
Moment.js Provides many date/time related functions. Ensure that the moment.js timezone functionality matches with the timezone used by the scheduler. Qubole uses Moment JS version 2.6.0. Moment.js
Moment-tokens Provides strftime formats Moment-tokens

Macros in API

Qubole supports macros in Command APIs, Command Template APIs, and Scheduler APIs. Macros in command template APIs apply only to the commands in the template.

In commands, macros are valid assignment statements containing the variables and its expression as: macros: [{"<variable>":<variable-expression>}, {..}].

You can add more than one variable.

Example to use a Macro in a Command API Call Request

Define a query

QUERY="SELECT stock_symbol, max(high), min(low), sum(volume) FROM daily_tick_data WHERE date1 >
'$sevendaysago$' AND date1 <= '$yesterday$' GROUP BY stock_symbol"

Submit a Hive Command through a API Request Call

curl  -i -X POST -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
 -d '{
       "query":"$QUERY", "command_type": "HiveCommand",
       "macros": [
            {"sevendaysago":"\"2017-01-10\""},
            {"yesterday":"\"2017-01-16\""},
      }' \
 "https://api.qubole.com/api/${V}/commands"

Caution

Notice that quotations are escaped when value of a macro is a string.

Macros in Scheduler

Macros in Scheduler describes in detail about the macros in scheduler. Scheduler API describes the scheduler APIs in QDS.

Macros in QDS UI

Currently Qubole supports macros for creating jobs/schedules on the Scheduler page. For more details, see Creating a New Schedule.