Save Pipeline Code

PUT /api/v1.2/pipelines/<pipeline_id>/save_code

Use this API to use a custom code or custom jar in a streaming pipeline. For more information about creating a streaming pipeline with custom code or custom JAR file by using the Quest UI, see Building Streaming Pipelines using Custom Jars or Code.

Required Role

The following users can make this API call:

  • Users who belong to the system-user or system-admin group.
  • Users who belong to a group associated with a role that allows submitting a command.
  • Users who belong to a role that allows the Quest resource.

See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold below are mandatory. Others are optional and have default values.

Parameter Description
create_type Type of code:3 for custom code or 2 for custom JAR.
code Spark application code written in scala or python. This parameter is applicable for custom code.
language Language of the code. Possible values are: scala or python.
jar_path S3 path of the spark application jar. This parameter is applicable for custom JAR.
main_class_name Main class of Spark application. This parameter is applicable for custom JAR.
user_arguments Program command line arguments.

Request API Syntax

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":<2-or-3>,
              "user_arguments":"<program-line-arguments>",
              "code":"<custom code>"
              "language":"<code-language>"
              "jar_path":"<s3-path>"
              "main_class_name":"<class-name>"

           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/<pipeline_id>/save_code"

Sample API Request

The following code shows how to use a custom code in a streaming pipeline.

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":3,
              "user_arguments":"arg0 arg1",
              "language":"scala"
              "code":"print("sc.version")"
           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/1/save_code"

The following code shows how to use a custom JAR file in a streaming pipeline.

curl -i -X PUT -H "X-AUTH-TOKEN: $AUTH_TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
-d `{
        "data":{
           "attributes":{
              "create_type":2,
              "user_arguments":"arg0 arg1",
              "jar_path":"s3://bucket/example.jar",
              "main_class_name":"org.apache.spark.SparkExampleProgram"
           }
        }
     }`\
"https://api.qubole.com/api/v1.2/pipelines/2/save_code"