Add Pipeline Properties

PUT /api/v1.2/pipelines/<pipeline_id>/properties

Use this API to add properties to a streaming pipeline in assisted mode.

Required Role

The following users can make this API call:

  • Users who belong to the system-user or system-admin group.
  • Users who belong to a group associated with a role that allows submitting a command.
  • Users who belong to a role that allows the Quest resource.

See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold below are mandatory. Others are optional and have default values.

Parameter Description
type Possible value: pipeline/properties
cluster_label Label of the spark cluster.
checkpoint_location Checkpoint location for the streaming query. This parameter is applicable only for the pipelines that are created in the assisted mode.
can_retry Boolean to specify whether to retry the pipeline in case of failure. Retry is attempted twice. Possible values: true or false.
output_mode Output mode for the streaming query. Possible values: append, update, and complete. This parameter is applicable only for the pipelines pipelines that are created in the assisted mode.
command_line_options Spark submit command line options. For example --conf spark.eventLog.compress=true.
trigger_interval Time in seconds after which a microbatch has to be triggered. For example, 2 seconds. This parameter is applicable only for the pipelines that are created in the assisted mode.

Request API Syntax

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{
    "data":{
       "attributes":{
          "cluster_label":"<cluster-label>",
          "can_retry":<true-or-false>,
          "command_line_options":"<command-option>",
          "checkpoint_location":"<s3-path>",
          "trigger_interval":<time-in-seconds>,
          "output_mode":"<output-mode>"
       },
       "type":"pipeline/properties"
    }
 }'\
"https://api.qubole.com/api/v1.2/pipelines/<pipeline_id>/properties"

Sample API Request

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d ' {
        "data":{
           "attributes":{
              "cluster_label":"spark",
              "can_retry":true,
              "command_line_options":"--conf spark.driver.extraLibraryPath=/usr/lib/hadoop2/lib/native",
              "checkpoint_location":"s3://bucket/checkpointPath",
              "trigger_interval":null,
              "output_mode":"Append"
           },
           "type":"pipeline/properties"
        }
     }`\
     "https://api.qubole.com/api/v1.2/pipelines/1/properties"