Debugging in DevOps Pipeline
Photo by Arif Riyanto on Unsplash
DevOps involves automating application deployment and focuses on easing the developer’s and operation team’s work.
The DevOps pipeline consists of many tools, such as GitHub, BitBucket, Jenkins, SonarQube, Ansible, Docker, Kubernetes, etc. All these tools are connected using various configurations. So if one tool fails, the entire process stops working. And if we don’t fix the issue within the fixed timeframe, it can create big problems. This brings us to the main focus of this essay—fixing DevOps pipelines with the help of debugging.
Debugging is the process of finding bugs in a software application and fixing them. Debugging an application is easy when there is one language or tool in play. However, with its multiple tools and languages, debugging in the DevOps pipeline is not an easy task. We need to know different debugging techniques to discover why the pipeline failed. Covering all the techniques for all the tools in one article is not possible. So here, I am focusing on a few main tools, like Jenkins, Docker, and Kubernetes, and discussing how to debug them using various techniques.
Debugging with Jenkins
Jenkins is used as a CI/CD (continuous integration and continuous deployment) tool. The main objective of Jenkins is to trigger a new job when a code is changed in a version control system (like GitHub or BitBucket) and package the application as a Docker image. It then deploys the Docker image to the Kubernetes cluster. Jenkins manages the entire deployment process.
From the Jenkins logs, we can easily know what was the problem in the DevOps pipeline. We can monitor Jenkins using various monitoring tools. Using the monitoring tools, we can also find out where a problem has occured. Jenkins has a sound logging system for each job. From the logs itself, we can easily understand what the problem is and why the problem occurred.
In Jenkins, we can create a multi-pipeline for different processes and set it up so that the second job runs only if the first job is successful in Jenkins. We can generate notification emails for the success or failure of a job. Most of the time, a Jenkins job fails due to a change in credentials in the remote server. So check the logs of each job and do the necessary steps to fix the bug.
Debug Docker Container
Docker is used to package the application as a portable image. By using Docker, we can create a container (running state of an application) from the image. Sometimes Docker may fail to generate an image or fail to run a container. Here, we are going to see a few different debugging techniques for Docker.
Assume we are trying to run a Docker container but it is not running. The Docker container can not start if the code used to start the application has some error. So, override the entry point command and start a container.
| docker run -i -d –entrypoint=bash container_id/container_name
#Use container name or container id #Example docker run -i -d –entrypoint=bash 893dr5tj9876 docker run -i -d –entrypoint=bash mysqldb |
Once the container is created, you can enter it into the container, and check the application to see why it failed to start.
| docker exec -it container_id/container_name bash
#Use container name or container id #Example docker exec -it mysqldb bash #or docker exec -it mysqldb /bin/bash #or docker exec -it mysqldb /bin/sh |
Get the Docker container logs and analyze them for any errors.
| docker logs container_id/container_name
#Example docker logs 893dr5tj9876 docker logs mysqldb |
Use the docker cp command to copy files from the docker container to the host machine and analyze them.
| docker cp container_id:/path/file.log /host_machine_path/
#Example docker cp container_id:/var/src/file.log /User/raja/Documents |
The above commands will help you find the problem in the Docker containers.
Debugging Kubernetes
Kubernetes is an open-source container orchestration tool that is used to scale Docker containers and manage multiple Docker containers. One might say Kubernetes is like an ocean. Using Kubernetes we can deploy containers, expose the application to external access, configure security, add a role for developers to access, and much more. Here, we are going to see a few useful commands to debug Kubernetes components.
Get the logs of the Kubernetes pod.
| kubectl logs pod_name
#Example kubectl logs nodejs |
Get the live logs of the Kubernetes pod.
| kubectl logs -f pod_name
#Example |
Get a detailed description of the logs using the describe command in the Kubernetes pod. You can similarly get services, secrets, config maps, etc.
| kubectl describe pod pod_name
#Example kubectl describe pod nodejs |
Enter into the Kubernetes pod and analyze it.
| kubectl exec -it pod_name bash #Example kubectl exec -it nodejs bash kubectl exec -it nodejs /bin/bash |
Use the cp command to copy files from the Kubernetes pod to the host machine.
| kubectl cp pod_name:/path/file_name /host_machine_path/ #Example kubectl cp nodejs:/var/src/error.log /User/raja/new_proj |
We can use the port forwarding concept to expose the application port to the host machine. Then, we can debug the application externally using the host machine IP address and host machine port.
| kubectl port-forward pod_name host_port:container_port #Example kubectl port-forward nodejs 8081:3000 |
Override the entry point command using the following syntax while creating a new pod from the Docker image. Here, command and args are used to override the entry point.
| — apiVersion: v1 kind: POD metadata: labels: app: testapp name: testapp spec: containers: – image: “test/ntestapp:1.1” name: nodejs command:[“echo”] args:[“Hey”] ports: – containerPort: 3000 |
Remote Debugging
Remote debugging lets you inspect the live running application from a local machine using the same code as the live running application. It is a great option if you are not able to reproduce the problem locally. Cloud Code is a VS code extension that lets you debug an application deployed to a Kubernetes cluster from a local machine.
Summary
The DevOps pipeline is used to automate application deployment. However, automation can fail for many reasons. This is largely because we incorporate many tools in DevOps application development. If one tool fails, the entire flow halts. So, we have to prepare ourselves to face any incident.
Here, I used a few tools in the DevOps process flow that are part of a custom setup. However, you can use third-party providers that offer managed cloud solutions such as AWS, GCP, and Azure. They even have a built-in DevOps pipeline. Debugging is a specialized skill. Especially when debugging a DevOps pipeline, you need different techniques to solve problems.
This article mainly focuses on Jenkins, Docker, and Kubernetes. We can find the bugs using the logs of each tool easily. Pay more attention to the logs and research why it has occurred. Make a note of all the possible bugs and solutions. It will help you and your teammates to solve the problem.
