HTTP Retry Policy

An HTTP request passes through multiple components such as DNS servers, load balancers, application servers, and so on before completion. Any component can generate an error during the transmission process. Qubole recommends adhering to a retry policy to increase the chances of a successful request.

Server-Side Errors (5xx series)

Qubole recommends you retry a request when the following server-side error codes are returned.

Status Code Error Code Retry?
503 Service Unavailable Yes
Other 5xx errors _ Yes (only for GET requests)

Client-Side Errors (4xx series)

Qubole recommends you retry a request when the following client-side error codes are returned.

Status Code Error Code Retry?
429 Too Many Requests Yes
449 Retry Yes (only for GET requests)
Other 4xx errors _ Yes (only for GET requests)

Retry a Failed Request with Exponential Backoff

If your request returns errors such as 429 (Too Many Requests) or 503 (Service Unavailable), you need to retry it after a delay to increase the chances of a successful response. Qubole recommends the following configurations for the number of retries and the interval between each retry. If you are using Qubole SDKs, these capabilities are applied by default.

Qubole SDKs adhere to the following values for these configurations.

Parameter Description Value
Number of recommended retries The maximum number of recommended retries. 7
Base delay interval The time interval after which the first retry is attempted. 10 seconds
Exponential backoff factor The factor by which the delay interval increases after the previous retry. For example, after the first try, the application waits for 20 seconds (2 * 10 seconds) before retrying the request. This increases to 40 seconds (2 * 20 seconds), and so on. 2

Sample Error Response

HTTP/1.1 429
Server: nginx
Date: Wed, 06 May 2020 11:25:50 GMT
Content-Length: 0
Connection: close
X-Qubole-Trace-Id: 4366736-1-1588764689