Automatically retry failed jobs?

[bford]

Is there a way to automatically retry a scheduled task that fails or send out notification that a job failed?

We’ve seen some instances where a job just needs to be run again and it will succeed, right now we’re scheduling some of our higher priority jobs twice the first time as a warm up, and the second time as the real deal, but would like to find a more programatic way to handle failed jobs.

[Eric_Lyons]

Hi @bford 

This is a great question. Generally, I would suggest only retrying schedules for specific errors like a timeout or cache cannot be found in results. The other errors are returned for a fairly specific reasons, so continually trying to send them can cause numerous errors and add overall load to the instance. But with that being said you could create a look in system activity with the scheduled plan explore.  (HOSTNAME/explore/system__activity/scheduled_plan) that collects the failed schedules (we can filter for status), download the results from that Look via the API and then try to send them again via the API schedule_plan_run_once_by_id. 

import pandas as pd
import looker_sdk
from looker_sdk import models40, methods
sdk = looker_sdk.init40()
me = sdk.me()
#get the look and write it as a csv because it is a bit easier to manage
look = sdk.run_look(look_id=2088,result_format="csv")
file = open('read.csv','w')
file.write(look)
file.close()
df = pd.read_csv("read.csv")
## create a df based on the look with a list of schedule plan ids where status = failure
#create a list of schedule plan ids
emails = df['Scheduled Plan ID'].tolist()
#loop through list of failed schedule plans and try to send them again based on id
i=0
while i < len(emails):
    details = sdk.scheduled_plan(scheduled_plan_id=emails)
    print(details.id)
    print(details.scheduled_plan_destination)
    print(details.name)
    schedule = sdk.scheduled_plan_run_once_by_id(scheduled_plan_id=details.id)
    i = i + 1

Please let me know if you have any questions.

Thanks,

Eric

[bford]

@Eric_Lyons Awesome thank you!  We’ll give that a try.