Add Pipeline Properties

PUT /api/v1.2/pipelines/<pipeline_id>/properties

Use this API to add properties to a streaming pipeline in assisted mode.

Required Role

The following users can make this API call:

• Users who belong to the system-user or system-admin group.
• Users who belong to a group associated with a role that allows submitting a command.
• Users who belong to a role that allows the Pipelines resource.

See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold below are mandatory. Others are optional and have default values.

Parameter	Description
type	Possible value: pipeline/properties
cluster_label	Label of the Spark streaming cluster.
checkpoint_location	Checkpoint location for the streaming query. This parameter is applicable only for the pipelines that are created in the assisted mode.
can_retry	Boolean to specify whether to retry the pipeline in case of failure. Retry is attempted twice. Possible values: true or false.
output_mode	Output mode for the streaming query. Possible values: append, update, and complete. This parameter is applicable only for the pipelines pipelines that are created in the assisted mode.
command_line_options	Spark submit command line options. For example --conf spark.eventLog.compress=true.
trigger_interval	Time in seconds after which a microbatch has to be triggered. For example, 2 seconds. This parameter is applicable only for the pipelines that are created in the assisted mode.

Request API Syntax

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{
    "data":{
       "attributes":{
          "cluster_label":"<cluster-label>",
          "can_retry":<true-or-false>,
          "command_line_options":"<command-option>",
          "checkpoint_location":"<s3-path>",
          "trigger_interval":<time-in-seconds>,
          "output_mode":"<output-mode>"
       },
       "type":"pipeline/properties"
    }
 }'\
"https://api.qubole.com/api/v1.2/pipelines/<pipeline_id>/properties"

Sample API Request

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d ' {
        "data":{
           "attributes":{
              "cluster_label":"spark",
              "can_retry":true,
              "command_line_options":"--conf spark.driver.extraLibraryPath=/usr/lib/hadoop2/lib/native",
              "checkpoint_location":"s3://bucket/checkpointPath",
              "trigger_interval":null,
              "output_mode":"Append"
           },
           "type":"pipeline/properties"
        }
     }`\
     "https://api.qubole.com/api/v1.2/pipelines/1/properties"