Add Pipeline Properties

PUT /api/v1.2/pipelines/<pipeline_id>/properties

Use this API to add properties to a streaming pipeline in assisted mode.

Required Role

The following users can make this API call:

  • Users who belong to the system-user or system-admin group.

  • Users who belong to a group associated with a role that allows submitting a command.

  • Users who belong to a role that allows the Pipelines resource.

See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold below are mandatory. Others are optional and have default values.

Parameter

Description

type

Possible value: pipeline/properties

cluster_label

Label of the Spark streaming cluster.

checkpoint_location

Checkpoint location for the streaming query. This parameter is applicable only for the pipelines that are created in the assisted mode.

can_retry

Boolean to specify whether to retry the pipeline in case of failure. Retry is attempted twice. Possible values: true or false.

output_mode

Output mode for the streaming query. Possible values: append, update, and complete. This parameter is applicable only for the pipelines pipelines that are created in the assisted mode.

command_line_options

Spark submit command line options. For example --conf spark.eventLog.compress=true.

trigger_interval

Time in seconds after which a microbatch has to be triggered. For example, 2 seconds. This parameter is applicable only for the pipelines that are created in the assisted mode.

Request API Syntax

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{
    "data":{
       "attributes":{
          "cluster_label":"<cluster-label>",
          "can_retry":<true-or-false>,
          "command_line_options":"<command-option>",
          "checkpoint_location":"<s3-path>",
          "trigger_interval":<time-in-seconds>,
          "output_mode":"<output-mode>"
       },
       "type":"pipeline/properties"
    }
 }'\
"https://api.qubole.com/api/v1.2/pipelines/<pipeline_id>/properties"

Sample API Request

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d ' {
        "data":{
           "attributes":{
              "cluster_label":"spark",
              "can_retry":true,
              "command_line_options":"--conf spark.driver.extraLibraryPath=/usr/lib/hadoop2/lib/native",
              "checkpoint_location":"s3://bucket/checkpointPath",
              "trigger_interval":null,
              "output_mode":"Append"
           },
           "type":"pipeline/properties"
        }
     }`\
     "https://api.qubole.com/api/v1.2/pipelines/1/properties"