Application lifecycle basics.
DevOps, CI/CD, and other trendy abbreviations.
Due to the relevance of the topic, nowadays market has a rich selection of tools intended to serve various stages of the application lifecycle. Technically it is not difficult for people to use Google to find information about this topic and follow documentation pages: but it is not that easy at the same time to get a general idea based on granular information. This article will try to fill in the voids and understand basic concepts about today's cloud applications' lifecycle.
This article may be interesting for
- junior engineers
- those whose daily work is indirectly related to software development
- those who have had a lot of exposure to legacy cloud applications in their careers are now interested in mastering the most common cloud environment practices.
In a general sense, speaking of a cloud application, we mean a piece of software that runs in a cloud environment. A cloud application often represents software products that run on several virtual machines or containers interconnected over the network. Together all the elements of an application perform some function or provide a service.
The concept of IaC came with the advent of the cloud culture. IaC, or Infrastructure as Code – is simply a source code of your cloud infrastructure or definition of the future state described with some special high-level programming or configuration language. The code contains a description of all the application elements typical for the cloud environment in which you'll deploy. These elements are VMs/pods, storage, network schemas, network security policies, software configuration, and custom elements. When well prepared - e.g., when it's clear and parameterized - it becomes reusable for different deployment environments (as Development, Testing, Staging, Production), making testing and release processes smoother, transparent, and, more importantly, reproducible. The choice of IaC tools (runtime) and, consequently, IaC language can depend on the cloud environment and is up to Application Architects' and DevOps Engineers' decision to evaluate what fits best their use case. There are no strict rules here, and even a mix of tools can be used to deploy one single application.
When we look at cloud applications' deployment and operations lifecycle as a code, they are also subject to most software development rules, no matter if it is a single web service or a set of 30 virtual machines with containers: all the challenges of software development are equally applicable to cloud application deployment as operation as-a-code.
That said, cloud applications go through a normal development cycle that is typical for any software application:
- Planing
- Design
- Development
- Testing
- Release
- Operations
Agile
When starting the development process, the team should agree on the methodology they will follow. There are several currently popular sets of principles and methods that help structure and optimize the development process.
For example, the Waterfall methodology assumed that a project's requirements do not change during the development process (or requirements stay the same for an extended period). Then, you split the process into sequential steps. The team proceeds to work on new tasks only after completing the previous phases, thus moving step by step towards implementing the project's full functionality. All stages of the development project, such as design, development, testing, are completed once in the Waterfall model.
In our company, the culture of Agile and the systems or framework derived from it (be it Scrum, Kanban, Scrumban, or others) converged, and most if not all teams in our company use some flavor of Agile in their daily work.
But what about all this? Agile proclaims principles to promote rapid and efficient development and minimize risks. From the process point of view, the main focus is to identify short development cycles: one cycle lasts from 2 weeks to 2 months. The code is released as often as possible: working closely with the customer simultaneously with development allows the team to quickly collect feedback and adjust to the requirements before it's too late - before the build and don't touch syndrome appears. Thus, working on a project turns from a linear task into a cycle of infinitely repeating iterations. Design – Develop – Test – Release – Monitor – Design.
The new steps bring new challenges. All of these steps require resources to maintain them. And in the context of cyclicality, this load will only increase. This problem is not at all new for software development. There are many mechanisms and tools which may help the application to go through all the steps mentioned above. And this is where the concept of CI/CD comes into play.
CI/CD
Historically the combined practices of CI/CD had formed earlier than the principles of Agile in its final form. But logically, it can be considered as a more substantive implementation of Agile ideas, aiming to shorten release times, reduce the number and cost of errors and improve product quality.
A few words about version control systems should make it easier to understand the CI/CD instruments.
Version control
The version control system is an integral part of the development process— both individual and collaboratively.
Distributed Version Control Systems are mostly used for collaborative development. They allow having a repository with a main copy of the code. Developers may create branches and local copies of the code stored on the server, make modifications, upload them to the server and merge changes into the main code. The system keeps a versioned history of all changes made to the code repository. Thus collaborative work on the project is ensured.
The most known VCSs are SVN (Apache Subversion), Mercurial, and Git.
In recent years Git has gained a lot of traction. Well-known source code control services as GitLab, GitHub, AWS CodeCommit all rely on it under the hood. In 2020, Bitbucket abandoned Mercurial in favor of Git. Azure DevOps services also offer their customers the ability to create Git repositories in addition to their centralized VCS - TFVC (Team Foundation Version Control).
Continuous Integration (CI)
When developers work independently on different parts of the application, using VCS, the integration stage, when all the work will be merged together, is final. It can delay the completion of the work unpredictably.
Continuous Integration, as a software development practice, consists of merging working copies into a common main development branch several times a day and performing frequent automated builds of the project to quickly identify potential defects and solve integration problems. The process assumes automatic testing of key indicators, so Continuous Testing may be considered as an integral sub-stream of the Continuous Integration process. The process itself may typically be started automatically on each code update event in the specific code branches.
The transition to Continuous Integration reduces the complexity of integration and makes it more predictable by early detection and elimination of errors and conflicts. The cost of error fixing reduces due to its early detection.
Continuous Delivery/Deployment (CD)
In general, the concept of CD has slightly more blurred boundaries than CI. According to the development cycle, integration and testing will be followed by a code release. Continuous Delivery is similar to Continuous Integration and represents the next step after it - the practice of automating the entire software release process. Some highly reactive teams that are ready to proceed effectively with faults may also want to consider Continuous Deployment, logically completing the automation of the development cycle with automatic deployment to the production environment. The others may want to have this stage under control and initiate it manually.
The most popular instruments now often combine the functionality of CI and CD. Tools such as Jenkins, AWS CodeBuild, Azure DevOps, GitHub Actions are commonly used. In our work, we mostly use GitLab for Source Code Management and GitLab CI/CD to proceed with the development cycle - for Continuous Delivery, in cases where Kubernetes is more present, teams may use ArgoCD or Flux2. As a GitOps instrument, these tools monitor a git-based repository for the changes to react and keep the application instance state equal to the target state (which is stored in the VCS).
However, it is easy to see that implementation and maintenance of CI/CD processes in the project would expand development tasks - and sometimes even go beyond them - cloud providers, operating systems, network protocols, and security, to name a few.
It extends developer's responsibilities, forcing them to understand the specifics of environments, corresponding tools and increasing the cost of the development at the same time. Projects and Managers should then ask the question: Is this change really worth the price?
DevOps
Not wanting to give up the advantages of agility and CI/CD in the context of rapidly developing cloud technologies, projects began to integrate administrators into the development teams, who were able to take a hit on the environment configuration topics. It also worth mentioning that up to this point, we have been talking about development only - however, the application's life journey only starts adding value to a customer or client with a product release (when the application is finally deployed): the code gets into the production environment, is explored, monitored, consumed by users.
Normally Operations teams handle the application's general production lifecycle. In the case of errors or some deviations (which are registered by a monitoring system), these boys and girls (the Operations Team) should take care of the application. However, the Operations Team is not involved in the development process of the application itself, they may have no sufficient information about it, and so unusual incidents will require the participation of the technical leader or developers. Сommunication between groups of different competencies slows down operations processes, even when steps are well documented: things become even less efficient when different companies are responsible for different stages.
This gap can be slightly covered by a stack of documentation; however, experience shows that even then, this will not help in defining a clear line between Ops and Dev teams' responsibilities, especially because these teams most often have conflicting interests: the Operations Team tends to strive to have a stable running application; whereas the Developer Team has its task to release changes to the application (e.g., features or bug fixes) which inevitably may increase chances of operation disruption.
Right here, at the place of the implicit intersection of the competencies, the DevOps culture and job title are born. But instead of an outdated approach with making borders, DevOps suggest removing them and extend Agile principles, applying them for both – development and operations, joining together the competencies of the developer, administrator, tester, and operator.
The DevOps team is responsible for the entire application lifecycle and its automation up to and including decommissioning.
Intersecting and borrowing ideas from Agile, DevOps also implies the use of CI/CD practices. But unlike the latter, it focuses less on tools and more on organizational changes. In particular, to support closer collaboration between different types of employees involved in the software delivery.
The DevOps model is quite extensive. It includes technical, organizational, and ideological aspects. Focuses on business and client values, team setup and motivation aspects, keeping space for continuous improvements.
It would be difficult to fit all the aspects of the DevOps model in one article, but there is no such task at the same time.
On the contrary, it is intended to show key aspects of cloud application lifecycle and software development process in broad strokes. And if you are interested to know more about the DevOps model, you can refer to books by DevOps enthusiasts like Gene Kim and his colleagues.
Conclusion
All the concepts and terms discussed above are closely intertwined by logical connections. They are designed to solve various problems that arise in the process of software development. Often without clear boundaries of definition, they have different areas of application.
Agile and scrum, as its tool, focuses on the approach to the development process as a whole, aiming to increase its efficiency, when CI/CD focuses on the implementation of the life cycle of the developed application and its automation. DevOps hovers over development and integration, solving problems of the next level. The culture pays attention to the roles of participants in the development and support processes of the application. It seeks to erase the boundaries between them, also aiming to improve efficiency and reduce costs.