Save Pipeline Code

PUT /api/v1.2/pipelines/<pipeline_id>/save_code

Use this API to use a custom code or custom jar in a streaming pipeline. For more information about creating a streaming pipeline with custom code or custom JAR file by using the Pipelines UI, see Building Streaming Pipelines using Custom Jars or Code.

Required Role

The following users can make this API call:

  • Users who belong to the system-user or system-admin group.

  • Users who belong to a group associated with a role that allows submitting a command.

  • Users who belong to a role that allows the Pipelines resource.

See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold below are mandatory. Others are optional and have default values.

Parameter

Description

create_type

Type of the pipeline to be created. Possible values: 2 for custom JAR or 3 for custom code.

code

Spark application code written in scala or python. This parameter is applicable for custom code.

language

Language of the code. Possible values are: scala or python.

jar_path

S3 path of the Spark application jar. This parameter is applicable for custom JAR.

main_class_name

Main class of the Spark application. This parameter is applicable for custom JAR.

user_arguments

Program command line arguments.

Request API Syntax

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":<2-or-3>,
              "user_arguments":"<program-line-arguments>",
              "code":"<custom code>"
              "language":"<code-language>"
              "jar_path":"<s3-path>"
              "main_class_name":"<class-name>"

           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/<pipeline_id>/save_code"

Sample API Request

The following code shows how to use a custom code in a streaming pipeline.

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":3,
              "user_arguments":"arg0 arg1",
              "language":"scala"
              "code":"print("sc.version")"
           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/1/save_code"

The following code shows how to use a custom JAR file in a streaming pipeline.

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":2,
              "user_arguments":"arg0 arg1",
              "jar_path":"s3://bucket/example.jar",
              "main_class_name":"org.apache.spark.SparkExampleProgram"
           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/2/save_code"