When we have issue with container and we are not sure which pod the container belongs to; we can find the pods details as below.
Use case : we have some zombie process running (defunct) but we are not sure which container is causing this zombie process.
Find the defunct processes
[root@node bin]# ps -ef |grep -v grep |grep -i defunct 1019710+ 15222 15581 0 05:30 ? 00:00:00 [sh]
1019710+ 15618 15581 0 05:30 ? 00:00:00 [sh] 1019710+ 37360 15581 0 Sep23 ? 00:00:00 [sh]
1019710+ 37569 15581 0 Sep23 ? 00:00:00 [sh] 1019710+ 37806 15581 0 Sep23 ? 00:00:00 [sh] 1019710+ 37939 15581 0 Sep23 ? 00:00:00 [git]
Find the docker image causing this zombies
We can use the systemd-cgls tool to find this; either run and search for the PID or grep for the PID.
[root@node bin]# systemd-cgls |grep -B4 15581 │ │ └─95146 /usr/bin/pod │ ├─kubepods-burstable-poda3a903e7_b912_11e8_8c1f_40f2e91e547d.slice
│ │ ├─docker-24695bc78b0f3908786fe00616df5sc023c9acc535aa4f53a8c5a04c96f840611.scope
│ │ │ ├─15492 /usr/bin/dumb-init -- /usr/libexec/s2i/run
│ │ │ └─15581 /usr/java/jdk1.8.0_181-amd64/bin/java -XX:+UseParallelGC -XX:…
You can see the info in docker line.
Find the container ID
(usually first 12 digit of that docker line)
[root@node bin]# docker ps |grep 24695bc78b0f
24695bc78b0f 10.221.246.15:5000/delivery-prod/jenkins-as-a-service@sha256:9c3b04e78c201ba50459e4806a5106ac326ec92c46ef5381f46fd2b0ed812ac4 "/usr/bin/dumb-init -" 8 days ago Up 8 days k8s_jenkins-as-a-service-abcd_jenkins-as-a-service-abcd-18-qj76w_test_proj_a3a903e7-b912-11e8-8c1f-40f2e91e547d_0
We can read the project and pod information from above line by splitting last string by “_”.
- jenkins-as-a-service-abcd – application
- jenkins-as-a-service-abcd-18-qj76w – pod
- test_proj – project
- a3a903e7-b912-11e8-8c1f-40f2e91e547d – uid
Find and verify pod details from master
[root@master ~]# oc get pods|grep jenkins-as-a-service-abcd
and verify
[root@master ~]# oc get pod jenkins-as-a-service-abcd-18-qj76w -o=json
(Switch to respective project as you need)