Are you sure SIGTERM is received by your application in Kubernetes?
Long time ago, I was doing resilience review for an application and figured that the shutdown hook was not executed by the application when the pod is deleted in Kubernetes, even though it had enough time to shutdown gracefully — defined by terminationGracePeriodSeconds
Only explanation is that the SIGTERM sent by kubelet is not actually received by the application. It had to do with the shell/exec form in dockerfile for ENTRYPOINTs. This is nothing new but this mistake is surprisingly common.
Understanding the difference between shell and exec forms
Both ENTRYPOINT/CMD in docker supports two forms,
- Shell form
ENTRYPOINT node app.js— Notice the main process (PID 1) is a shell process which starts PID 7. So SIGTERM sent to your process is eaten up at PID 1 itself and not passed on to the child process — i.e. your application.
$ docker exec 4675d ps x
PID TTY STAT TIME COMMAND
1 ? Ss 0:00 /bin/sh -c node app.js
7 ? Sl 0:00 node app.js
13 ? Rs+ 0:00 ps x- Exec form
ENTRYPOINT ["node", "app.js"]— runs the node process directly (not inside shell)
$ docker exec 4675d ps x
PID TTY STAT TIME COMMAND
1 ? Ssl 0:00 node app.js
12 ? Rs 0:00 ps xYou cannot pass $variables in Docker’s ENTRYPOINT/CMD exec form
ENTRYPOINT bin/project $PORTS — works
ENTRYPOINT ["bin/project", “$PORTS"] — $PORTS does not get resolved since it is not run via shell command.
Of course you can fix this by enclosing the command in a shell script and executing it using exec. This shell script can resolve variables.
Other way is to use Kubernetes command. Example :
env:
- name: MESSAGE
value: "hello world"
command: ["/bin/echo"]
args: ["$(MESSAGE)"]References
- https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
- https://github.com/moby/moby/issues/18492
- https://github.com/moby/moby/issues/18492#issuecomment-376207145 — interesting usecase.
- Kubernetes in action — under section “UNDERSTANDING THE DIFFERENCE BETWEEN THE SHELL AND EXEC FORMS”
- Useful commands to verify this for your application :
// get a shell to the running container
kubectl -n <namespace> exec -it <pod_name> -- bash// install this to execute ps command.
apt-get update && apt-get install procps// if you see sh -c in the list it means you have used shell form. you can double check your deployment file as well, to be sure.
ps -x// stream the logs in a different console
kubectl -n <namespace> -f logs <pod_name> // This should send sigterm to your process. you can watch the logs for any shutdown hook logic that you expect to get triggered on termination.
kubectl -n <namespace> delete pod <pod_name>
- As best practice, define shutdown hooks relevant to your application. This whole exercise becomes irrelevant otherwise.
