Troubleshooting with kubectl

Kubectl is a command-line tool used to run commands against Kubernetes clusters.

With ARTESCA, kubectl commands can be used to help users identify and analyze the global status of the K8s cluster, but not to modify or delete any resources.

Make sure to follow these recommendations:

  • Use the artesca-os user with the sudo command to manage kubectl (because the root user is only available on the console).

  • Use a tmux or screen terminal when performing maintenance operations (such as an upgrade).

  • Export the admin.conf file to authenticate and access the K8s cluster to avoid any error. You can set the argument in your profile.

    [artesca-os@bootstrap ~]$

    export KUBECONFIG=/etc/kubernetes/admin.conf

  • Always specify the namespace using the -n <namespace> or -A flag in pod commands to avoid a No resources found in default namespace error message.

The following sections offer a list of commands you can run to get information about nodes, events, and pods, and troubleshoot potential issues on your cluster.

Basic Troubleshooting Commands

Check Node Status

Run this command to check the status of nodes. The -o wide flag displays a wide output containing additional information such as hostnames and IPs.

[artesca-os@bootstrap ~]$

sudo kubectl get nodes -o wide
Output Example
NAME              STATUS   ROLES                              AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                            KERNEL-VERSION                 CONTAINER-RUNTIME
pme-arte6demo-1   Ready    bootstrap,etcd,infra,master,node           324d   v1.27.6   10.0.11.220   <none>       …
pme-arte6demo-2   Ready    etcd,infra,master,node                     324d   v1.27.6   10.0.1.149    <none>       …
pme-arte6demo-3   Ready    etcd,infra,master,node                     324d   v1.27.6   10.0.1.132    <none>       …
pme-arte6demo-4   Ready    node                                       114d   v1.27.6   10.0.1.88     <none>

Status

Description

Ready

The node is running as expected.

NotReady

The node is currently unavailable.

Check systemctl status kubelet on the server to see if the service is running well. Restart the service if needed.

SchedulingDisabled

The node cannot start a new pod because of a lack of resources.

A common error is full /var/lib. You can see in the node’s description the disk-pressure``or ``DiskPressure information.

This can also happen after an upgrade or when the node has not been uncordon. Refer to Node and Cluster Maintenance for more information.

Check Cluster Activity

Events records the activity of a cluster, run this command to check for errors.

[artesca-os@bootstrap ~]$

sudo kubectl events -A
Output Example
NAMESPACE         LAST SEEN               TYPE      REASON                  OBJECT                                               MESSAGE
zenko             60m                     Normal    Pulled                  Pod/pods-balancer-28759970-nwnts                     Container image "metalk8s-registry-from-config.invalid/metalk8s-127.0.3/metalk8s-utils:127.0.3" already present on machine
zenko             60m                     Normal    SuccessfulCreate        CronJob/pods-balancer                                Created job pods-balancer-28759970
zenko             60m                     Normal    SuccessfulCreate        Job/pods-balancer-28759970                           Created pod

Check Pod Status

Run this command to check the status of pods.

[artesca-os@bootstrap ~]$

sudo kubectl get pods -A
Output Example
NAMESPACE             NAME                                                              READY   STATUS      RESTARTS        AGE
artesca-auth          artesca-keycloak-0                                                1/1     Running     5 (129m ago)    8d
artesca-auth          artesca-keycloak-1                                                1/1     Running     5 (129m ago)    8d

Status

Description

Running

Pod is running as expected.

CrashLoopBackOff

The pod is stuck in a restart loop.

Check Pod Details for more information.

Pending

Pod has not been scheduled for running.

Check Pod Details for more information.

Evicted

Pod is terminated. This is usually due to a lack of resources.

Check Pod Details for more information.

CreateContainerError

The pod cannot be scheduled because the name is reserved for another pod.

Refer to Resolve Unschedulable Pods Due to Reserved Names for more information.

ImagePullBackOff

The pod image has not been successfully retrieved.

Check Pod Details for more information.

Advanced Troubleshooting Commands

Check Pod Details

Run the describe command to get detailed information on a specific pod.

[artesca-os@bootstrap ~]$

sudo kubectl describe pod <pod_name> -n <namespace>
Example
NAMESPACE             NAME                                                              READY   STATUS      RESTARTS       AGE     IP               NODE              NOMINATED NODE   READINESS GATES
xcore                 artesca-storage-service-ab159db9-ds-0                             0/1     Pending     0              5m23s   <none>           <none>            <none>           <none>
...
Type     Reason            Age                    From               Message
----     ------            ----                   ----               -------
Warning  FailedScheduling  2m33s (x3 over 7m55s)  default-scheduler  0/4 nodes are available: 1 node(s) didn't find available persistent volumes to bind. preemption: 0/4 nodes are available: 1 Preemption
is not helpful for scheduling, 3 No preemption victims found for incoming pod..

Or, run the get pods command and grep excluding the Running and Completed statuses from the output.

[artesca-os@bootstrap ~]$

sudo kubectl get pods -o wide -A| grep -v -e Running -e Completed
Output Example
NAMESPACE             NAME                                                              READY   STATUS      RESTARTS       AGE     IP               NODE              NOMINATED NODE   READINESS GATES
xcore                 artesca-storage-service-ab159db9-ds-0                             0/1     Pending     0              5m23s   <none>           <none>            <none>           <none>

Important

Some pods might be running, but their containers are not running. Confirm that all pods and all containers are running.

[artesca-os@bootstrap ~]$

sudo kubectl get pods -o wide -A| egrep -v -e "1/1|2/2|3/3" -e Completed

Extract Pod Details

Use the get -o yaml command to extract the pods details.

[artesca-os@bootstrap ~]$

sudo kubectl get pod <pod_name> -n <namespace> -o yaml

View Pod Logs

Run the logs command to view the logs of a specific pod.

[artesca-os@bootstrap ~]$

sudo kubectl logs <pod_name> -n <namespace>

Get Pod Labels

  1. Run the show-labels command to get the labels of a specific pod.

    [artesca-os@bootstrap ~]$

    sudo kubectl get pods -n <namespace> --show-labels

  2. Run the labels command with the -l flag to get the pods with a specific label. Refer to the kubernetes documentation for more information on label selectors.

    [artesca-os@bootstrap ~]$

    sudo kubectl get pods -n <namespace> -l <label>

View Replicaset Logs

Run the logs command to view the logs of a specific Replicaset.

[artesca-os@bootstrap ~]$

sudo kubectl logs replicaset/<replicaset_name> -n <namespace>

View Statefulset Logs

Run the logs command to view the logs of a specific Statefulset.

[artesca-os@bootstrap ~]$

sudo kubectl logs  sts/<sts_name> -n <namespace>

View Deployment Logs

Run the logs command to view the logs of a specific Deployment.

[artesca-os@bootstrap ~]$

sudo kubectl logs  deployment/<deployment_name> -n <namespace>

Tip

For more information on kubectl, see the official Kubernetes documentation.