Save Pipeline Code

PUT /api/v1.2/pipelines/<pipeline_id>/save_code

Use this API to use a custom code or custom jar in a streaming pipeline. For more information about creating a streaming pipeline with custom code or custom JAR file by using the Pipelines UI, see Building Streaming Pipelines using Custom Jars or Code.

Required Role

The following users can make this API call:

• Users who belong to the system-user or system-admin group.
• Users who belong to a group associated with a role that allows submitting a command.
• Users who belong to a role that allows the Pipelines resource.

See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold below are mandatory. Others are optional and have default values.

Parameter	Description
create_type	Type of the pipeline to be created. Possible values: 2 for custom JAR or 3 for custom code.
code	Spark application code written in scala or python. This parameter is applicable for custom code.
language	Language of the code. Possible values are: scala or python.
jar_path	S3 path of the Spark application jar. This parameter is applicable for custom JAR.
main_class_name	Main class of the Spark application. This parameter is applicable for custom JAR.
user_arguments	Program command line arguments.

Request API Syntax

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":<2-or-3>,
              "user_arguments":"<program-line-arguments>",
              "code":"<custom code>"
              "language":"<code-language>"
              "jar_path":"<s3-path>"
              "main_class_name":"<class-name>"

           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/<pipeline_id>/save_code"

Sample API Request

The following code shows how to use a custom code in a streaming pipeline.

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":3,
              "user_arguments":"arg0 arg1",
              "language":"scala"
              "code":"print("sc.version")"
           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/1/save_code"

The following code shows how to use a custom JAR file in a streaming pipeline.

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":2,
              "user_arguments":"arg0 arg1",
              "jar_path":"s3://bucket/example.jar",
              "main_class_name":"org.apache.spark.SparkExampleProgram"
           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/2/save_code"