Configurations

Important fields

1. .spec.jobTemplate.spec.template.spec.restartPolicy: How to handle when a container in a pod. Note that this doesn’t mean a job restarts or not (it’s commented on this github issue).

• onFailure: Restart a container in the same pod when it fails.
• Never: Restart a container with new pod.

1. .spec.concurrencyPolicy: How to handle concurrent jobs

• Allow (Default): Allows concurrent runs
• Forbid: Does not allow concurrent runs
• Replace: Replace old job with new job if previous job hasn’t finished when it’s time to schedule new job

1. .spec.successfulJobsHistoryLimit, .spec.failedJobsHistoryLimit: How many pods is kept after it succeeded or failed
2. .spec.startingDeadlineSeconds: How long jobs keep trying to start. Also, see a pitfall section.

Pitfalls

StartingDeadlineSeconds

A cronjob stops starting jobs if it “misses” jobs more than 100 times in a specific period related with .spec.startingDeadlineSeconds. The details are explained in here.

Here is the brief explanation, though they might be wrong or incomplete.

• If startingDeadlineSeconds isn’t set, the number of missed jobs is counted since last scheduled time. And if it exceeds more than 100 times, then jobs won’t start. If startingDeadlineseconds is set, then missed jobs counted in last startingDeadlineSeconds.
• The number of missed jobs includes when a new job cannot run because a concurrencyPolicy is Forbid and previous job is still running.

Other pitfalls

1. Jobs should be “idempotent”, because the same jobs might be created more than once, or no job might be created.
2. CronJobs may run more than once even if .spec.parallelism = 1, .spec.completions = 1, and .spec.template.spec.restartPolicy = “Never”

• See official page for more details.

1. CronJobs usually fail to start more than 30 or 60 seconds (in 2018).

• See this slide (Japanese) for more details.

1. Workaround is required for cronjobs with istio sidecars. There is an option to update manifests of istio sidecar to stop the sidecar after corresponding job container stops, or disable injecting istio sidecar.

• See this GitHub issues for workarounds to insert istio sidecar for cronjobs
• See this GitHub issues to support a sidecar as a kubernetes 1st citizen. After this issue is solved, no workaround won’t be required anymore.

kubectl commands

1. How to run a job from cronjob: kubectl create job --from=cronjob/[cronjob name] [job name]

References

1. Kubernetes documentation: Running Automated Tasks with a CronJob
2. Kubernetes documentation: Jobs
3. How we learned to improve Kubernetes CronJobs at Scale (Part 1 of 2)
4. What does Kubernetes cronjob’s startingDeadlineSeconds exactly mean?