In this article, I will talk about debugging and troubleshooting Kubernetes pods using ephemeral containers.
The simplest way to debug pods is to exec into the problematic pods and try to troubleshoot what is happening. This is a simple approach but it has many drawbacks.
- The running application pods may not have all the required tools to troubleshoot an existing issue.
- If you want to perform some actions that require additional permissions, you will need to restart all pods for the currently running application pods to add the new requirements.
- They are introducing security risks by adding debugging tools inside the main docker image, also if container permissions are elevated.
so, let’s explore another way to debug pods.
Ephemeral containers are useful for interactive troubleshooting when
kubectl exec is insufficient because a container has crashed or a container image doesn’t include debugging utilities, such as distroless images, or the running pods don’t have the required privileges for debugging.
The main idea behind ephemeral containers is that K8S adds a new container with a selected custom image to an existing pod without the need for restarting this pod. This new container share can share many resources from the target containers which are,
- Linux network namespace
- Linux process namespace
- Access to shared volumes
- Access to k8s node
I will give an example for each of these use cases.
Before starting the demo, you need to have a k8s cluster with version 1.23. I recommend using
kind but you can use any other provisioner.
so let’s start by creating a cluster for our demo
Creating a new kind of cluster is simple as running the command
kind create cluster
Once the cluster is created, you need to verify that it is up and accessible
All of our operations will be executed from master
kind node so we will need to access it by
docker exec -it <kind-container-id> bash
We will assume that we have an Nginx deployment that we want to debug, so let’s create an Nginx deployment with one replicator. This will be done by running this command
kubectl create deployment nginx --image=nginx
Troubleshooting network activity requires sharing network namespace, This is the default Linux namespace when you attach an ephemeral container to a running pod.
let’s create our first ephemeral container, I will use
knicolaka/netshoot as an image for the new ephemeral container. This image contains many troubleshooting tools like
tcpdump and strace
kubectl debug --it pod-name --image=<ephemeral-container> -- command
So let’s confirm that both containers share the same Linux namespace. Open a new shell to the master node, and run this command
systemd-cgls -u kubelet-kubepods-besteffort.slice
From the above example, we can get the main process IDs for both containers
- 2612 -> main process ID for the ephemeral container
- 2259 -> main process ID for the Nginx container
Now, let’s check all Linux namespaces for each of these processes
From the previous screenshot, we found that both processes have the same Linux network namespace id.
Now let’s dump network packets for the Nginx container from the ephemeral containers.
From the ephemeral container shell, run this command
tcpdump -n port 80
Now, try to send some requests to this pod from the k8s master node
Now, if you go to the ephemeral container terminal, you will find the dump of TCP packets is printed to the output:
We finished our 1st demo and now, we can capture the network packets from the ephemeral container.
Let’s go to the second use case.
Our next use case for ephemeral containers is tracing a process running in a container from another container.
To achieve this, we will need:
- The two containers must share the same Linux process namespace.
- The ephemeral container must have a Linux capability
Sharing a Linux process namespace is can be done easily when creating the ephemeral container by adding an additional argument
kubectk debug -it <pod-name > --image=nicolaka/netshoot --target <container-name> -- bash
As you can see from the previous screenshot:
- In order to share the process namespace, we can just add an additional command argument
- From the ephemeral container, we can see all running processes from
- we can’t trace
nginxprocess as the ephemeral containers doesn’t have the required permission to send
ptracesystem call. This system call is used by strace command to pause the Linux process to record each system call that
nginxsends to the kernel.
How can we fix this? Unfortunately, I didn’t find a way to pass extra permissions to the ephemeral container from
kubectl command. So we will construct and send an HTTP request to kube API server without the use of
Now, you can
strace without getting permission denied.
For this example, I added a permission to do
SYS_PTRACE. But it depends on the debugger you are using, or simply, you can give the ephemeral container privileged access. So you don’t need to worry about which system calls you need to allow.
Another thing to mention, you can access the filesystem for
nginx container from the ephemeral container. The root file system is under
Let’s see if we can access
nginx config from the ephemeral container.
Ok, let’s now start the last selected use case for ephemeral containers
Sometimes, you need access to k8s node, but you don’t have ssh access or console access to the node.
You can access the node by using an ephemeral container
kubectl debug node/<node-name> -it --image=<image-name>
When creating a debugging session on a node, keep in mind that:
kubectl debugautomatically generates the name of the new Pod based on the name of the Node.
- The container runs in the host IPC, Network, and PID namespaces.
- The root filesystem of the Node will be mounted at
If you want the root file system of the ephemeral container is the same as the node, You will just need to