When your app relies on making API calls to external resources, you should expect that there will be network issues from time to time and you should prepare for them.

While these will be rare relative to the number of requests you're making, handling them early on could really save you a lot of headache in the future as your app usage grows. Many of these things can be easily handled with a simple retry decorator.

If you're using Celery, it also has a way to retry the entire task, but using a retry decorator for certain functions or class methods within the task would still help.

For example, all the apps we've built for Highview Apps so far use Celery. But we have tasks that make multiple API calls to Shopify to retrieve the user's data because the API can only return a certain number of records in the response at a time. One of the endpoints, for instance, can only return 250 records at a time, but if the user has 100,000 records that would be 400 API calls in one task. If one call fails, the task can retry but it will have to start over. So in this case, it's better to also be able to retry the specific function or method that makes the API call so you'll only be retrying that one call that failed.

Below is a nice retry decorator I found on this site. I just changed one line of code to make it Python 3 compatible (basically had to change the print statement to a function).

import time
from functools import wraps


def retry(exceptions, tries=4, delay=3, backoff=2, logger=None):
    """
    Retry calling the decorated function using an exponential backoff.

    Args:
        exceptions: The exception to check. may be a tuple of
            exceptions to check.
        tries: Number of times to try (not retry) before giving up.
        delay: Initial delay between retries in seconds.
        backoff: Backoff multiplier (e.g. value of 2 will double the delay
            each retry).
        logger: Logger to use. If None, print.
    """
    def deco_retry(f):

        @wraps(f)
        def f_retry(*args, **kwargs):
            mtries, mdelay = tries, delay
            while mtries > 1:
                try:
                    return f(*args, **kwargs)
                except exceptions as e:
                    msg = '{}, Retrying in {} seconds...'.format(e, mdelay)
                    if logger:
                        logger.warning(msg)
                    else:
                        print(msg)
                    time.sleep(mdelay)
                    mtries -= 1
                    mdelay *= backoff
            return f(*args, **kwargs)

        return f_retry  # true decorator

    return deco_retry

Source

This decorator allows you to retry a function/method based on the exception raised in that function/method.

A few notes

• The decorator uses exponential backoff so each retry attempt takes longer.
• It will work on both functions and class methods which is made possible with the *args and **kwargs parameters.
• You can pass in multiple exceptions as a tuple.

Usage

Retry the request if it throws a Timeout exception.

import requests

@retry(requests.exceptions.Timeout)
def call_github():
    return requests.get('https://api.github.com/events')

Retry the request if it throws a Timeout or ConnectionError.

import requests
from requests.exceptions import ConnectionError, Timeout

@retry((ConnectionError, Timeout))
def call_github():
    return requests.get('https://api.github.com/events')

Retry the request 5 times if any exception occurs.

import requests

@retry(Exception, tries=6)
def call_github():
    return requests.get('https://api.github.com/events')

You can also pass in a logger object to the decorator so it will be logged as a warning with a message that it will retry. If all retries failed, the exception is simply raised.