Resolve Job Failures in Kubernetes

Problem Description

You receive a KubeJobFailed alert which means that some jobs did not complete successfully.

Problem Resolution

You need to investigate the job failure and delete the failed job to clear the alert.

1. Take note of the <namespace> and <job> fields in the alert’s description.

   Alert description example

   Job zenko/artesca-data-ops-count-items-12345678 Failed to complete
   

   In this example, the namespace is zenko and the job is artesca-data-ops-count-items-12345678.

2. Fetch the pod name to investigate its logs.

   Tip

   Replace <namespace> and <job> with the namespace and job shown in the alert.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -n <namespace> --selector=job-name=<job>
   

   Note

   If you see the message No resources found in <namespace> namespace, the job has already been deleted. This error is expected in some cases, for example during an upgrade. In this case, skip the next step and proceed to the step Delete the failed job.

3. Investigate the job failure.

   Tip

   Replace <namespace> and <pod> with the namespace and pod.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf logs -n <namespace> <pod>
   

4. Delete the failed job.

   Tip

   Replace <namespace> and <job> with the namespace and job.

   [artesca-os@bootstrap ~]$
   

   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf delete job -n <namespace> <job>