Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew

Question

I've created and pushed a cron job to deployment, but when I see it running in OpenShift, I get the following error message:

Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.

From what I understand by this, is that a job failed to run. But I don't understand why it is failing. Why isn't that logged somewhere? - or if it is, where can I find it?

The CronJob controller will keep trying to start a job according to the most recent schedule, but keeps failing and obviously it has done so >100 times.

I've checked the syntax of my cron job, which doesn't give any errors. Also if there are any syntax messages, I'm not even allowed to push.

Anyone know what's wrong?

my Cron Job:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-cjob
  labels:
    job-name: my-cjob
spec:
  schedule: "*/5 * * * *" 
  # activeDeadlineSeconds: 180 # 3 min <<- should this help and why?      
  jobTemplate:
      spec:
        template:         
          metadata:
            name: my-cjob
            labels:
              job-name: my-cjob
          spec:
            containers:
            - name: my-cjob
              image: my-image-name
            restartPolicy: OnFailure

Or should I be using startingDeadlineSeconds? Anyone who has hit this error message and found a solution?

Update as according to comment

When running kubectl get cronjob I get the following:

NAME           SCHEDULE      SUSPEND   ACTIVE    LAST SCHEDULE   AGE
my-cjob        */5 * * * *   False     0         <none>          2d

When running kubectl logs my-cjob I get the following:

Error from server (NotFound): pods "my-cjob" not found

When running kubectl describe cronjob my-cjob I get the following:

Error from server (NotFound): the server could not find the requested resource

When running kubectl logs <cronjob-pod-name> I get many lines o code... Very difficult for me to understand and sort out..

When running kubectl describe pod <cronjob-pod-name> I also get a lot, but this is way easier to sort. Anything specific?

Running kubectl get events I get a lot, but I think this is the related one:

LAST SEEN   FIRST SEEN   COUNT     NAME                                            KIND                    SUBOBJECT                                 TYPE      REASON              SOURCE                                      MESSAGE
1h          1h           2         xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx             Pod                     spec.containers{apiproxy}                 Warning   Unhealthy           kubelet, xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx   Liveness probe failed: Get http://xxxx/xxxx: dial tcp xxxx:8080: connect: connection refused

Answer 1

Setting the startingDeadlineSeconds to 180 fixed the problem + removing the spec.template.metadata.labels.

Answer 2

I suspended my workload then resumed it after quite a while and saw the same error. Isn't it a bug because I triggered the suspend action on purpose anytime between suspend and resume should NOT be counted against missing starting.

Answer 3

The root cause for this issue:

For every CronJob, the CronJob Controller checks how many schedules it missed in the duration from its last scheduled time until now. If there are more than 100 missed schedules, then it does not start the job and logs the error.^1

A CronJob is counted as missed if it has failed to be created at its scheduled time. For example, if concurrencyPolicy is set to Forbid and a CronJob was attempted to be scheduled when there was a previous schedule still running, then it would count as missed.^1

The simplest solution I can think of is recreating the cronjob to clean the missed schedules.